Methods and systems for food preparation in a robotic cooking kitchen

ABSTRACT

The present disclosure is directed to methods, computer program products, and computer systems for instructing a robot to prepare a food dish by replacing the human chef&#39;s movements and actions. Monitoring a human chef is carried out in an instrumented application-specific setting, a standardized robotic kitchen in this instance, and involves using sensors and computers to watch, monitor, record and interpret the motions and actions of the human chef, in order to develop a robot-executable set of commands robust to variations and changes in the environment, capable of allowing a robotic or automated system in a robotic kitchen to prepare the same dish to the standards and quality as the dish prepared by the human chef.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.14/627,900, filed on 20 Feb. 2015, now issued as U.S. Pat. No.9,815,191, which claims priority to U.S. Provisional Application Ser.No. 62/116,563 entitled “Method and System for Food Preparation in aRobotic Cooking Kitchen,” filed on 16 Feb. 2015, U.S. ProvisionalApplication Ser. No. 62/113,516 entitled “Method and System for FoodPreparation in a Robotic Cooking Kitchen,” filed on 8 Feb. 2015, U.S.Provisional Application Ser. No. 62/109,051 entitled “Method and Systemfor Food Preparation in a Robotic Cooking Kitchen,” filed on 28 Jan.2015, U.S. Provisional Application Ser. No. 62/104,680 entitled “Methodand System for Robotic Cooking Kitchen,” filed on 16 Jan. 2015, U.S.Provisional Application Ser. No. 62/090,310 entitled “Method and Systemfor Robotic Cooking Kitchen,” filed on 10 Dec. 2014, U.S. ProvisionalApplication Ser. No. 62/083,195 entitled “Method and System for RoboticCooking Kitchen,” filed on 22 Nov. 2014, U.S. Provisional ApplicationSer. No. 62/073,846 entitled “Method and System for Robotic CookingKitchen,” filed on 31 Oct. 2014, U.S. Provisional Application Ser.62/055,799 entitled “Method and System for Robotic Cooking Kitchen,”filed on 26 Sep. 2014, U.S. Provisional Application Ser. No. 62/044,677,entitled “Method and System for Robotic Cooking Kitchen,” filed on 2Sep. 2014, U.S. Provisional Application Ser. No. 62/024,948 entitled“Method and System for Robotic Cooking Kitchen,” filed on 15 Jul. 2014,U.S. Provisional Application Ser. No. 62/013,691 entitled “Method andSystem for Robotic Cooking Kitchen,” filed on 18 Jun. 2014, U.S.Provisional Application Ser. No. 62/013,502 entitled “Method and Systemfor Robotic Cooking Kitchen,” filed on 17 Jun. 2014, U.S. ProvisionalApplication Ser. No. 62/013,190 entitled “Method and System for RoboticCooking Kitchen,” filed on 17 Jun. 2014, U.S. Provisional ApplicationSer. No. 61/990,431 entitled “Method and System for Robotic CookingKitchen,” filed on 8 May 2014, U.S. Provisional Application Ser. No.61/987,406 entitled “Method and System for Robotic Cooking Kitchen,”filed on 1 May 2014, U.S. Provisional Application Ser. No. 61/953,930entitled “Method and System for Robotic Cooking Kitchen,” filed on 16Mar. 2014, and U.S. Provisional Application Ser. No. 61/942,559 entitled“Method and System for Robotic Cooking Kitchen,” filed on 20 Feb. 2014,the disclosures of which are incorporated herein by reference in theirentireties.

BACKGROUND Technical Field

The present invention relates generally to the interdisciplinary fieldsof robotics and artificial intelligence, more particularly tocomputerized robotic food preparation systems for food preparation bydigitizing the food preparation process of professional andnon-professional chef dishes and subsequently replicating a chef'scooking movements, processes and techniques with real-time electronicadjustments.

Background Art

Research and development in robotics have been undertaken for decadesbut the progress has been mostly in the heavy industrial applicationslike automobile manufacturing automation or military applications.Simple robotics systems have been designed for the consumer markets buthave largely not seen a wide application in the home-consumer roboticsspace thus far. With advances in technology, combined with a populationwith higher incomes, the market may be ripe to create opportunities fortechnological advances to improve people's lives. Robotics has continuedto improve automation technology with enhanced artificial intelligenceand emulation of human skills and tasks in many forms.

The notion of robots replacing humans in certain areas and executingtasks humans would typically perform is an ideology in continuousevolution since robots first were developed in the 1970s. Manufacturingsectors have long used robots in teach-playback mode, where the robot istaught, via pendant or offline fixed-trajectory generation and download,which motions to copy continuously and without alteration or deviation.Companies have taken the pre-programmed trajectory-execution ofcomputer-taught trajectories and robot motion-playback into suchapplication domains as mixing drinks, welding or painting cars, andothers. However, all of these conventional applications use a 1:1computer-to-robot or tech-playback principle that is intended to haveonly the robot faithfully execute the motion-commands, which is almostalways following a taught/pre-computed trajectory without deviation.

Gastronomy is an art of eating well, where a gourmet recipe blendssubtly high quality ingredients and flavor appealing to all our senses.Gourmet cooking follows rules based on techniques that can be veryelaborate, requiring expertise and technique, and lengthy training insome cases. In the past few years, demand for gourmet food has growntremendously because of fast rising incomes and a generational shift inculinary awareness. However, diners still need to visit a certainrestaurant or venue for gourmet dishes made by a favored chef. It wouldbe rather advantageous to see a chef preparing your favorite dish livein action or experience a dish preparation reminiscent of a childhooddish made by your grandmother.

Accordingly, it would be desirable to have a system and method to have achef's gourmet dish made and served conveniently to consumers in theirown home(s), without the necessity to travel to each restaurant aroundthe world to enjoy specific gourmet dishes.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure are directed to methods, computerprogram products, and computer systems of a robotic apparatus withrobotic instructions replicating a food dish with substantially the sameresult as if the chef had prepared the food dish. In a first embodiment,the robotic apparatus in a standardized robotic kitchen comprises tworobotic arms and hands, which replicate the precise movements of a chefin the same sequence (or substantially the same sequence) and the sametiming (or substantially the same timing) to prepare a food dish basedon a previously recorded software file (a recipe-script) of the chef'sprecise movements in preparing the same food dish. In a secondembodiment, a computer-controlled cooking apparatus prepares a food dishbased on a sensory-curve, such as temperature over time, which waspreviously recorded in a software file where the chef prepared the samefood dish with the cooking apparatus with sensors for which a computerrecorded the sensor values over time when the chef previously preparedthe food dish on the cooking apparatus fitted with sensors. In a thirdembodiment, the kitchen apparatus comprises the robotic arms in thefirst embodiment and the cooking apparatus with sensors in the secondembodiment to prepare a dish that combines both the robotic arms and oneor more sensory curves, where the robotic arms are capable ofquality-checking a food dish during the cooking process, for suchcharacteristics as taste, smell, and appearance, allowing for anycooking adjustments to the preparation steps of the food dish. In afourth embodiment, the kitchen apparatus comprises a food storage systemwith computer-controlled containers and container identifiers forstoring and supplying ingredients for a user to prepare a food dish byfollowing a chef's cooking instructions. In a fifth embodiment, arobotic cooking kitchen comprises a robot with arms and a kitchenapparatus in which the robot moves around the kitchen apparatus toprepare a food dish by emulating a chef's precise cooking movements,including possible real-time modifications/adaptations to thepreparation process defined in the recipe-script.

A robotic cooking engine comprises detection, recording, and chefemulation cooking movements, controlling significant parameters, such astemperature and time, and processing the execution with designatedappliances, equipment, and tools, thereby reproducing a gourmet dishthat tastes identical to the same dish prepared by a chef and served ata specific and convenient time. In one embodiment, a robotic cookingengine provides robotic arms for replicating a chef's identicalmovements with the same ingredients and techniques to produce anidentical tasting dish.

The underlying motivation of the present disclosure centers aroundhumans being monitored with sensors during their natural execution of anactivity and then being able to use monitoring-sensors,capturing-sensors, computers and software to generate information andcommands to replicate the human activity using one or more roboticand/or automated systems. While one can conceive of multiple suchactivities (e.g. cooking, painting, playing an instrument, etc.), oneaspect of the present disclosure is directed to the cooking of a meal;in essence a robotic meal preparation application. Monitoring the humanis carried out in an instrumented application-specific setting (astandardized kitchen in this case), and involves using sensors andcomputers to watch, monitor, record and interpret the motions andactions of a human chef, in order to develop a robot-executable set ofcommands robust to variations and changes in the environment, capable ofallowing a robotic or automated system in a robotic kitchen to preparethe same dish to the standards and quality as the dish prepared by thehuman chef.

The use of multimodal sensing systems is the means by which thenecessary raw data is collected. Sensors capable of collecting andproviding such data include environment and geometrical sensors, such astwo- (cameras, etc.) and three-dimensional (lasers, sonar, etc.)sensors, as well as human motion-capture systems (human-worncamera-targets, instrumented suits/exoskeletons, instrumented gloves,etc.), as well as instrumented (sensors) and powered (actuators)equipment used during recipe creation and execution (instrumentedappliances, cooking-equipment, tools, ingredient dispensers, etc.). Allthis data is collected by one or more distributed/central computers andprocessed by a variety of software processes. The algorithms willprocess and abstract the data to the point that a human and acomputer-controlled robotic kitchen can understand the activities,tasks, actions, equipment, ingredients and methods and processes used bythe human, including replication of key skills of a particular chef. Theraw data is processed by one or more software abstraction engines tocreate a recipe-script that is both human-readable and, through furtherprocessing, machine-understandable and machine-executable, spelling outall actions and motions for all steps of a particular recipe that arobotic kitchen would have to execute. These commands range incomplexity from controlling individual joints, to a particularjoint-motion profile over time, to abstracted levels of commands, withlower-level motion-execution commands embedded therein, associated withspecific steps in a recipe. Abstracted motion-commands (e.g. “crack anegg into the pan”, “sear to a golden color on both sides”, etc.) can begenerated from the raw data, and refined and optimized through amultitude of iterative learning processes, carried out live and/oroff-line, allowing the robotic kitchen systems to successfully deal withmeasurement-uncertainties, ingredient variations, etc., enabling complex(adaptive) mini-manipulation motions using fingered-hands mounted torobot-arms and wrists, based on fairly abstracted/high-level commands(e.g. “grab the pot by the handle”, “pour out the contents”, “grab thespoon off the countertop and stir the soup”, etc.).

The ability to create machine-executable command sequences, nowcontained within digital files capable of being shared/transmitted,allowing any robotic kitchen to execute them, opens up the option toexecute the dish-preparation steps anywhere at any time. Hence it allowsfor the option to buy/sell recipes online, allowing users to access anddistribute recipes on a per-use or subscription basis.

The replication of a dish prepared by a human is performed by a robotickitchen, which is in essence a standardized replica of the instrumentedkitchen used by the human chef during the creation of the dish, exceptthat the human's actions are now carried out by a set of robotic armsand handtheed by computer-monitored and computer-controllableappliances, equipment, tools, dispensers, etc. The degree ofdish-replication fidelity will thus be tightly tied to the degree towhich the robotic kitchen is a replica of the kitchen (and all itselements and ingredients) in which the human chef was observed whilepreparing the dish.

Broadly stated, there may be provided a computer-implemented methodoperating on a robotic apparatus, comprising an electronic descriptionof one or more food dishes, including the recipes for making each fooddish from ingredients by a chef; for each food dish, sensing a sequenceof observations of a chef's movements by a plurality of robotic sensorsas the chef prepares the food dish using ingredients and kitchenequipment; detecting in the sequence of observations mini-manipulationscorresponding to a sequence of movements carried out in each stage ofpreparing a particular food dish; transforming the sensed sequence ofobservations into computer readable instructions for controlling arobotic apparatus capable of performing the sequences ofmini-manipulations; storing at least the sequence of instructions formini-manipulations on electronic media for each food dish, wherein thesequence of mini-manipulations for each food dish is stored as arespective electronic record; transmitting the respective electronicrecord for a food dish to a robotic apparatus capable of replicating thesequence of stored mini-manipulations, corresponding to the originalactions of the chef; and executing the sequence of instructions formini-manipulations for a particular food dish by the robotic apparatus,thereby obtaining substantially the same result as the original fooddish prepared by the chef, wherein executing the instructions includessensing properties of the ingredients used in preparing the food dish.

Advantageously, the robotic apparatus in a standardized robotic kitchenhas the capabilities to prepare a wide array of cuisines from around theworld through a global network and database access, as compared to achef who may specialize in one type of cuisine. The standardized robotickitchen also is able to capture and record one of your favorite fooddishes for replication by the robotic apparatus whenever you like toenjoy the food dish without the repetitive process of laboring toprepare the same dish over and over again.

The structures and methods of the present invention are disclosed in thedetailed description below. This summary does not purport to define theinvention. The invention is defined by the claims. These and otherembodiments, features, aspects, and advantages of the invention willbecome better understood with regard to the following description,appended claims, and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with respect to specific embodimentsthereof, and reference will be made to the drawings, in which:

FIG. 1 is a system diagram illustrating an overall robotic foodpreparation kitchen with hardware and software in accordance with thepresent invention.

FIG. 2 is a system diagram illustrating a first embodiment of a foodrobot cooking system that includes a chef studio system and a householdrobotic kitchen system in accordance with the present invention.

FIG. 3 is system diagram illustrating one embodiment of the standardizedrobotic kitchen for preparing a dish by replicating a chef's recipeprocess, techniques and movements in accordance with the presentinvention.

FIG. 4 is a system diagram illustrating one embodiment of a robotic foodpreparation engine for use with the computer in the chef studio systemand the household robotic kitchen system in accordance with the presentinvention.

FIG. 5A is a block diagram illustrating a chef studio recipe-creationprocess in accordance with the present invention; FIG. 5B is blockdiagram illustrating one embodiment of a standardized teach/playbackrobotic kitchen in accordance with the present invention; FIG. 5C is ablock diagram illustrating one embodiment of a recipe script generationand abstraction engine in accordance with the present invention; andFIG. 5D is a block diagram illustrating software elements forobject-manipulation in the standardized robotic kitchen in accordancewith the present invention.

FIG. 6 is a block diagram illustrating a multimodal sensing and softwareengine architecture in accordance with the present invention.

FIG. 7A is a block diagram illustrating a standardized robotic kitchenmodule used by a chef in accordance with the present invention; FIG. 7Bis a block diagram illustrating the standardized robotic kitchen modulewith a pair of robotic arms and hands in accordance with the presentinvention; FIG. 7C is a block diagram illustrating one embodiment of aphysical layout of the standardized robotic kitchen module used by achef in accordance with the present invention; FIG. 7D is a blockdiagram illustrating one embodiment of a physical layout of thestandardized robotic kitchen module used by a pair of robotic arms andhands in accordance with the present invention; and FIG. 7E is a blockdiagram depicting the stepwise flow and methods to ensure that there arecontrol or verification points during the recipe replication processbased on the recipe-script when executed by the standardized robotickitchen in accordance with the present invention.

FIG. 8A is a block diagram illustrating one embodiment of a conversionalgorithm module between the chef movements and the robotic mirrormovements in accordance with the present invention; FIG. 8B is a blockdiagram illustrating a pair of gloves with sensors worn by the chef 49for capturing and transmitting the chef's movements; FIG. 8C is a blockdiagram illustrating robotic cooking execution based on the capturedsensory data from the chef's gloves in accordance with the presentinvention; FIG. 8D is a graphical diagram illustrating dynamicallystable and dynamically unstable curves relative to equilibrium; FIG. 8Eis a sequence diagram illustrating the process of food preparation thatrequires a sequence of steps that are referred to as stages inaccordance with the present invention; FIG. 8F is a graphical diagramillustrating the probability of overall success as a function of thenumber of stages to prepare a food dish in accordance with the presentinvention; and FIG. 8G is a block diagram illustrating the execution ofa recipe with multi-stage robotic food preparation withmini-manipulations and action primitives.

FIG. 9A is a block diagram illustrating an example of robotic hand andwrist with haptic vibration, sonar, and camera sensors for detecting andmoving a kitchen tool, an object, or a piece of kitchen equipment inaccordance with the present invention; FIG. 9B is a block diagramillustrating a pan-tilt head with sensor camera coupled to a pair ofrobotic arms and hands for operation in the standardized robotic kitchenin accordance with the present invention; FIG. 9C is a block diagramillustrating sensor cameras on the robotic wrists for operation in thestandardized robotic kitchen in accordance with the present invention;FIG. 9D is a block diagram illustrating an eye-in-hand on the robotichands for operation in the standardized robotic kitchen in accordancewith the present invention; and FIGS. 9E-I are pictorial diagramsillustrating aspects of deformable palm in a robotic hand in accordancewith the present invention.

FIG. 10A is block diagram illustrating examples of chef recordingdevices which a chef wears in the robotic kitchen environment forrecording and capturing his or her movements during the food preparationprocess for a specific recipe; and FIG. 10B is a flow diagramillustrating one embodiment of the process in evaluating the capturingof a chef's motions with robot poses, motions and forces in accordancewith the present invention.

FIG. 11 is block diagram illustrating a side view of a robotic armembodiment for use in the household robotic kitchen system in accordancewith the present invention.

FIGS. 12A-C are block diagrams illustrating one embodiment of a kitchenhandle for use with the robotic hand with the palm in accordance withthe present invention.

FIG. 13 is a pictorial diagram illustrating an example robotic hand withtactile sensors and distributed pressure sensors in accordance with thepresent invention.

FIG. 14 is a pictorial diagram illustrating an example of a sensingcostume for a chef to wear at the robotic cooking studio in accordancewith the present invention.

FIGS. 15A-B are pictorial diagrams illustrating one embodiment of athree-fingered haptic glove with sensors for food preparation by thechef and an example a three-fingered robotic hand with sensors inaccordance with the present invention.

FIG. 16 is a block diagram illustrating the creation module of amini-manipulation database library and the execution module of themini-manipulation database library in accordance with the presentinvention.

FIG. 17A is a block diagram illustrating a sensing glove used by a chefto execute standardized operating movements in accordance with thepresent invention; and FIG. 17B is a block diagram illustrating adatabase of standardized operating movements in the robotic kitchenmodule in accordance with the present invention.

FIG. 18A is a graphical diagram illustrating that each of the robotichand coated with a artificial human-like soft-skin glove in accordancewith the present invention; FIG. 18B is a block diagram illustratingrobotic hands coated with artificial human-like skin gloves to executehigh-level mini-manipulations based on a library database ofmini-manipulations, which have been predefined and stored in the librarydatabase, in accordance with the present invention; FIG. 18C is agraphical diagram illustrating three types of taxonomy of manipulationactions for food preparation in accordance with the present invention;FIG. 18D is a flow diagram illustrating one embodiment on taxonomy ofmanipulation actions for food preparation in accordance with the presentinvention; FIG. 18E is a block diagram illustrating one example of theinterplay and interactions between a robotic arm and a robotic hand inaccordance with the present invention; and FIG. 18F is a block diagramFIG. 18F is a block diagram illustrating the robotic hand uses thestandardized kitchen handle that is attachable to a cookware head andthe robotic arm attachable to kitchen ware in accordance with thepresent invention.

FIG. 19 is a block diagram illustrating the creation of amini-manipulation that results in cracking an egg with knife an examplein accordance with the present invention.

FIG. 20 is a block diagram illustrating an example of recipe executionfor a mini-manipulation with real-time adjustment in accordance with thepresent invention.

FIG. 21 is a flow diagram illustrating the software process to capture achef's food preparation movements in a standardized kitchen module inaccordance with the present invention.

FIG. 22 is a flow diagram illustrating the software process for foodpreparation by robotic apparatus in the robotic standardized kitchenmodule in accordance with the present invention.

FIG. 23 is a flow diagram illustrating one embodiment of the softwareprocess for creating, testing, and validating, and storing the variousparameter combinations for, a mini-manipulation system in accordancewith the present invention.

FIG. 24 is a flow diagram illustrating one embodiment of the softwareprocess for creating the tasks for a mini-manipulation system inaccordance with the present invention.

FIG. 25 is a flow diagram illustrating the process of assigning andutilizing a library of standardized kitchen tools, standardized objects,and standardized equipment in a standardized robotic kitchen inaccordance with the present invention.

FIG. 26 is a flow diagram illustrating the process of identifying anon-standardized object with three-dimensional modeling in accordancewith the present invention.

FIG. 27 is a flow diagram illustrating the process for testing andlearning of mini-manipulations in accordance with the present invention.

FIG. 28 is a flow diagram illustrating the process for robotic armsquality control and alignment function process in accordance with thepresent invention.

FIG. 29 is a table illustrating a database library structure ofmini-manipulations objects for use in the standardized robotic kitchenin accordance with the present invention.

FIG. 30 is a table illustrating a database library structure ofstandardized objects for use in the standardized robotic kitchen inaccordance with the present invention.

FIG. 31 is a pictorial diagram illustrating a robotic hand forconducting quality check of fish in accordance with the presentinvention.

FIG. 32 is a pictorial diagram illustrating a robotic sensor head forconducting quality check in a bowl in accordance with the presentinvention.

FIG. 33 is a pictorial diagram illustrating a detection device orcontainer with a sensor for determining the freshness and quality offood in accordance with the present invention.

FIG. 34 is a system diagram illustrating an online analysis system fordetermining the freshness and quality of food in accordance with thepresent invention.

FIG. 35 is a block diagram illustrating pre-filled containers withprogrammable dispenser control in accordance with the present invention.

FIG. 36 is a block diagram illustrating a recipe system structure foruse in the standardized robotic kitchen in accordance with the presentinvention.

FIGS. 37A-C are block diagrams illustrating recipe search menus for usein the standardized robotic kitchen in accordance with the presentinvention; FIG. 37D is a screen shot of a menu with option to create andsubmit a recipe in accordance with the present invention; FIGS. 37E-Mare flow diagrams illustrating one embodiment of the food preparationuser interface with functional capabilities including a recipe filter,an ingredient filter, an equipment filter, an account and social networkaccess, a personal partner page, a shopping cart page, and theinformation on the purchased recipe, registration setting, create arecipe in accordance with the present invention; and FIG. 37N-V arescreen shots of various graphical user interface and menu options inaccordance with the present invention.

FIG. 38 is a block diagram illustrating a recipe search menu byselecting fields for use in the standardized robotic kitchen inaccordance with the present invention.

FIG. 39 is a block diagram illustrating the standardized robotic kitchenwith an augmented sensor for three-dimensional tracking and referencedata generation in accordance with the present invention.

FIG. 40 is a block diagram illustrating the standardized robotic kitchenwith multiple sensors for creating real-time three-dimensional modelingin accordance with the present invention.

FIGS. 41A-L are block diagrams illustrating the various embodiments andfeatures of the standardized robotic kitchen in accordance with thepresent invention.

FIG. 42A is block diagram illustrating a top plan view of thestandardized robotic kitchen in accordance with the present invention;and FIG. 42B is a block diagram illustrating a perspective plan view ofthe standardized robotic kitchen in accordance with the presentinvention.

FIGS. 43A-B are block diagrams illustrating a first embodiment of thekitchen module frame with automatic transparent doors in thestandardized robotic kitchen in accordance with the present invention;and FIGS. 43C-F are block diagrams illustrating screen shots and asample kitchen module specification in the standardized robotic kitchenin accordance with the present invention.

FIGS. 44A-B are block diagrams illustrating a second embodiment of thekitchen module frame with automatic transparent doors in thestandardized robotic kitchen in accordance with the present invention.

FIG. 45 is a block diagram illustrating the standardized robotic kitchenwith a telescopic actuator in accordance with the present invention.

FIG. 46A is a block diagram illustrating a front view of thestandardized robotic kitchen with a pair of fixed robotic arms with nomoving railings in accordance with the present invention; FIG. 46B is ablock diagram illustrating an angular view of the standardized robotickitchen with a pair of fixed robotic arms with no moving railings inaccordance with the present invention; and FIGS. 46C-G are blockdiagrams illustrating examples of various dimensions in the standardizedrobotic kitchen with a pair of fixed robotic arms with no movingrailings in accordance with the present invention.

FIG. 47 is a block diagram illustrating a program storage system for usewith the standardized robotic kitchen in accordance with the presentinvention.

FIG. 48 is a block diagram illustrating an elevation view of the programstorage system for use with the standardized robotic kitchen inaccordance with the present invention.

FIG. 49 is a block diagram illustrating an elevation view of ingredientaccess containers for use with the standardized robotic kitchen inaccordance with the present invention.

FIG. 50 is a block diagram illustrating an ingredient quality monitoringdashboard associated with ingredient access containers for use with thestandardized robotic kitchen in accordance with the present invention.

FIG. 51 is a table illustrating a database library of recipe parametersin accordance with the present invention.

FIG. 52 is a flow diagram illustrating the process of one embodiment ofrecording a chef's food preparation process in accordance with thepresent invention.

FIG. 53 is a flow diagram illustrating the process of one embodiment ofa robotic apparatus preparing a food dish in accordance with the presentinvention.

FIG. 54 is a flow diagram illustrating the process of one embodiment inthe quality and function adjustment in obtaining the same orsubstantially the same result in a food dish preparation by a roboticrelative to a chef in accordance with the present invention.

FIG. 55 is a flow diagram illustrating a first embodiment in the processof the robotic kitchen preparing a dish by replicating a chef'smovements from a recorded software file in a robotic kitchen inaccordance with the present invention.

FIG. 56 is a flow diagram illustrating the process of storage check-inand identification in the robotic kitchen in accordance with the presentinvention.

FIG. 57 is a flow diagram illustrating the process of storage check-outand cooking preparation in the robotic kitchen in accordance with thepresent invention.

FIG. 58 is a flow diagram illustrating one embodiment of an automatedpre-cooking preparation process in the robotic kitchen in accordancewith the present invention.

FIG. 59 is a flow diagram illustrating one embodiment of a recipe designand scripting process in the robotic kitchen in accordance with thepresent invention.

FIG. 60 is a flow diagram illustrating a subscription model for the userto purchase the robotic food preparation recipe in accordance with thepresent invention.

FIGS. 61A-B are flow diagrams illustrating the process of a recipesearch and purchase subscription for a recipe commerce platform from aportal in accordance with the present invention.

FIG. 62 is a flow diagram illustrating the creation of a robotic cookingrecipe app on an app platform in accordance with the present invention.

FIG. 63 is a flow diagram illustrating the process of a user search,purchase, and subscription for a cooking recipe in accordance with thepresent invention.

FIGS. 64A-B are block diagrams illustrating an example of a predefinedrecipe search criterion in accordance with the present invention.

FIG. 65 is a block diagram illustrating some pre-defined containers inthe robotic kitchen in accordance with the present invention.

FIG. 66 is a block diagram illustrating a first embodiment of a roboticrestaurant kitchen module configured in a rectangular layout withmultiple pairs of robotic hands for simultaneous food preparationprocessing in accordance with the present invention.

FIG. 67 is a block diagram illustrating a second embodiment of a roboticrestaurant kitchen module configured in a U-shape layout with multiplepairs of robotic hands for simultaneous food preparation processing inaccordance with the present invention.

FIG. 68 is a block diagram illustrating a second embodiment of therobotic food preparation system with sensory cookware and curves inaccordance with the present invention.

FIG. 69 is a block diagram illustrating some physical elements of arobotic food preparation system in the second embodiment in accordancewith the present invention.

FIG. 70 is a block diagram illustrating sensory cookware for a (smart)pan with real-time temperature sensors for use in the second embodimentin accordance with the present invention.

FIG. 71 is a graphical diagram illustrating the recorded temperaturecurve with multiple data points from the different sensors of thesensory cookware in the chef studio in accordance with the presentinvention.

FIG. 72 is a graphical diagram illustrating the recorded temperature andhumidity curves from the sensory cookware in the chef studio fortransmission to an operating control unit in accordance with the presentinvention.

FIG. 73 is a block diagram illustrating sensory cookware for cookingbased on the data from a temperature curve for different zones on a panin accordance with the present invention.

FIG. 74 is a block diagram illustrating sensory cookware of a (smart)oven with real-time temperature and humidity sensors for use in thesecond embodiment in accordance with the present invention.

FIG. 75 is a block diagram illustrating a sensory cookware for a (smart)charcoal grill with real-time temperature sensors for use in the secondembodiment in accordance with the present invention.

FIG. 76 is a block diagram illustrating sensory cookware for a (smart)faucet with speed, temperature and power control functions for use inthe second embodiment in accordance with the present invention.

FIG. 77 is a block diagram illustrating a top plan view of a robotickitchen with sensory cookware in the second embodiment in accordancewith the present invention.

FIG. 78 is a block diagram illustrating a perspective view of a robotickitchen with sensory cookware in the second embodiment in accordancewith the present invention.

FIG. 79 is a flow diagram illustrating a second embodiment in theprocess of the robotic kitchen preparing a dish from one or moreprevious recorded parameter curves in a robotic kitchen in accordancewith the present invention.

FIG. 80 is a flow diagram illustrating the second embodiment of therobotic food preparation system by capturing a chef's cooking processwith sensory cookware in accordance with the present invention.

FIG. 81 is a flow diagram illustrating the second embodiment of therobotic food preparation system by replicating a chef's cooking processwith sensory cookware in accordance with the present invention.

FIG. 82 is a block diagram illustrating a third embodiment of therobotic food preparation kitchen with a cooking operating controlmodule, and a command and visual monitoring module in accordance withthe present invention.

FIG. 83 is a block diagram illustrating a top plan view in the thirdembodiment of the robotic food preparation kitchen with robotic arm andhand motions in accordance with the present invention.

FIG. 84 is a block diagram illustrating a perspective view in the thirdembodiment of the robotic food preparation kitchen with robotic arm andhand motions in accordance with the present invention.

FIG. 85 is a block diagram illustrating a top plan view in the thirdembodiment of the robotic food preparation kitchen with a command andvisual monitoring device in accordance with the present invention.

FIG. 86 is a block diagram illustrating a perspective view in the thirdembodiment of the robotic food preparation kitchen with a command andvisual monitoring device in accordance with the present invention.

FIG. 87A is a block diagram illustrating a fourth embodiment of therobotic food preparation kitchen with a robot in accordance with thepresent invention; FIG. 87B is a block diagram illustrating a top planview in the fourth embodiment of the robotic food preparation kitchenwith the humanoid robot in accordance with the present invention; andFIG. 87C is a block diagram illustrating a perspective plan view in thefourth embodiment of the robotic food preparation kitchen with thehumanoid robot in accordance with the present invention.

FIG. 88 is a block diagram illustrating a robotic human-emulatorelectronic intellectual property (IP) library in accordance with thepresent invention.

FIG. 89 is a block diagram illustrating a robotic human emotionrecognition engine in accordance with the present invention.

FIG. 90 is a flow diagram illustrating the process of a robotic humanemotion engine in accordance with the present invention.

FIGS. 91A-91C are flow diagrams illustrating the process of comparing aperson's emotional profile against a population of emotional profileswith hormones, pheromones and other parameters in accordance with thepresent invention.

FIG. 92A is a block diagram illustrating the emotional detection andanalysis of a person's emotional state by monitoring a set of hormones,a set of pheromones, and other key parameters in accordance with thepresent invention; and FIG. 92B is a block diagram illustrating a robotassessing and learning about a person's emotional behavior in accordancewith the present invention.

FIG. 93 is a block diagram illustrating a port device implanted in aperson to detect and record the person's emotional profile in accordancewith the present invention.

FIG. 94A is a block diagram illustrating a robotic human intelligenceengine in accordance with the present invention; and FIG. 94B is a flowdiagram illustrating the process of a robotic human intelligence enginein accordance with the present invention.

FIG. 95A is a block diagram illustrating a robotic painting system inaccordance with the present invention; FIG. 95B is a block diagramillustrating the various components of a robotic painting system inaccordance with the present invention; and FIG. 95C is a block diagramillustrating the robotic human-painting-skill replication engine inaccordance with the present invention.

FIG. 96A is a flow diagram illustrating the recording process of anartist at a painting studio in accordance with the present invention;and FIG. 96B is a flow diagram illustrating the replication process by arobotic painting system in accordance with the present invention.

FIG. 97A is block diagram illustrating an embodiment of a musicianreplication engine in accordance with the present invention; and FIG.97B is block diagram illustrating the process of the musicianreplication engine in accordance with the present invention.

FIG. 98 is block diagram illustrating an embodiment of a nursingreplication engine in accordance with the present invention.

FIGS. 99A-B are flow diagrams illustrating the process of the nursingreplication engine in accordance with the present invention.

FIG. 100 is a block diagram illustrating an example of a computer deviceon which computer-executable instructions to perform the roboticmethodologies discussed herein may be installed and executed.

DETAILED DESCRIPTION

A description of structural embodiments and methods of the presentinvention is provided with reference to FIGS. 1-100. It is to beunderstood that there is no intention to limit the invention to thespecifically disclosed embodiments but that the invention may bepracticed using other features, elements, methods, and embodiments. Likeelements in various embodiments are commonly referred to with likereference numerals.

The following definitions apply to the elements and steps describedherein. These terms may likewise be expanded upon.

Abstracted Data—refers to the abstracted recipe of utility formachine-execution which has many other data-elements that a machineneeds to know for proper execution and replication. This so-calledmeta-data, or additional data corresponding to a particular step in thecooking process, whether it be direct sensor-data (clock-time,water-temperature, camera-image, utensil or ingredient used, etc.) ordata generated through interpretation or abstraction of larger data-sets(such as a 3-dimensional range cloud from a laser used to extract thelocation and types of objects in the image, overlaid with texture andcolor maps from a camera-picture, etc.), is time-stamped and used by therobotic kitchen to set, control and monitor all processes and associatedmethods and equipment needed at every point in time as it steps throughthe sequence of steps in the recipe.

Abstracted Recipe—refers to a representation of a chef's recipe, which ahuman knows as represented by the use of certain ingredients, in certainsequences, prepared and combined through a sequence of processes andmethods as well as skills of the human chef. An abstracted recipe usedby a machine for execution in an automated way requires different typesof classifications and sequences. While the overall steps carried outare identical to those of the human chef, the abstracted recipe ofutility to the robotic kitchen requires that additional meta-data be apart of every step in the recipe. Such meta-data includes the cookingtime, variables such as temperature (and its variations over time),oven-setting, tool/equipment used, etc. Basically a machine-executablerecipe-script needs to have all possible measured variables of import tothe cooking process (all measured and stored while the human chef waspreparing the recipe in the chef studio) correlated to time, bothoverall and that within each process-step of the cooking-sequence. Hencethe abstracted recipe is a representation of the cooking steps mappedinto a machine-readable representation or domain, which takes therequired process from the human-domain to that of themachine-understandable and machine-executable domain through a set oflogical abstraction steps.

Acceleration—refers to the maximum rate of speed-change at which arobotic arm can accelerate around an axis or along a space-trajectoryover a short distance.

Accuracy—refers to how closely a robot can reach a commanded position.Accuracy is determined by the difference between the absolute positionof the robot compared to the commanded position. Accuracy can beimproved, adjusted, or calibrated with external sensing such as sensorson a robotic hand or a real-time three-dimensional model using multiple(multi-mode) sensors.

Action Primitive—In one embodiment, the term refers to an indivisiblerobotic action, such as moving the robotic apparatus from location X1 tolocation X2, or sensing the distance from an object for food preparationwithout necessarily obtaining a functional outcome. In anotherembodiment, the term refers to an indivisible robotic action in asequence of one or more such units for accomplishing amini-manipulation. These are two aspects of the same definition.

Automated Dosage System—refers to dosage containers in a standardizedkitchen module where a particular size of food chemical compounds (suchas salt, sugar, pepper, spice, any kind of liquids, such as water, oil,essences, ketchup, etc.) that is released upon application.

Automated Storage and Delivery System—refers to storage containers in astandardized kitchen module that maintain a specific temperature andhumidity for storing food; each storage container is assigned a code(e.g., a bar code) for the robotic kitchen to identify and retrievalwhere a particular storage container delivers the food contents storedtherein.

Data Cloud—refers to a collection of sensor or data-based numericalmeasurement values from a particular space (three-dimensionallaser/acoustic range measurement, RGB-values from a camera image, etc.)collected at certain intervals and aggregated based on a multitude ofrelationships, such as time, location, etc.

Degree of Freedom (“DOF”)—refers to a defined mode and/or direction inwhich a mechanical device or system can move. The number of degrees offreedom is equal to the total number of independent displacements oraspects of motion. The total number of degrees of freedom is doubled fortwo robotic arms.

Edge Detection—refers to a software-based computer program(s) capable ofidentifying the edges of multiple objects that may be overlapping in atwo-dimensional-image of a camera yet successfully identifying theirboundaries to aid in object identification and planning for grasping andhandling.

Equilibrium Value—refers to the target position of a robotic appendage,such as a robotic arm where the forces acting upon it are inequilibrium, i.e. there is no net force and thus no net movement.

Execution Sequence Planner—refers to a software-based computerprogram(s) capable of creating a sequence of execution scripts orcommands for one or more elements or systems capable of being computercontrolled, such as arm(s), dispensers, appliances, etc.

Food Execution Fidelity—refers to a robotic kitchen which is intended toreplicate the recipe-script generated in the chef studio by watching andmeasuring and understanding the steps and variables and methods andprocesses of the human chef, thereby trying to emulate his/hertechniques and skills. The fidelity of how close the execution of thedish-preparation comes to that of the human-chef is measured by howclose the robotically-prepared dish resembles the human-prepared dish asmeasured by a variety of subjective elements, such as consistency,color, taste, etc. The notion is that, the more closely the dishprepared by the robotic kitchen is to that prepared by the human chef,the higher the fidelity of the replication process.

Food Preparation Stage (also referred to as “Cooking stage”)—refers to acombination, either sequential or in parallel, of one or moremini-manipulations including action primitives, and computerinstructions for controlling the various kitchen equipment andappliances in the standardized kitchen module; one or more foodpreparation stages collectively represent the entire food preparationprocess for a particular recipe.

Geometric Reasoning—refers to a software-based computer program(s)capable of using two-dimensional (2D)/three-dimensional (3D) surface-and/or volumetric data to reason as to the actual shape and size of aparticular volume; the ability to determine or utilize boundaryinformation also allows for inferences as to the start end of aparticular geometric element and the number present (in an image ormodel).

Grasp Reasoning—refers to a software-based computer program(s) capableof relying on geometric and physical reasoning to plan a multi-contact(point/area/volume) contact-interaction between a robotic end-effector(gripper, link, etc.), or even tools/utensils held by the end-effector,so as to successfully and stably contact, grasp and hold the object inorder to manipulate it in three-dimensional space.

Hardware Automation Device—Fixed process device capable of executingpre-programmed steps in succession without the ability to modify any ofthem; such devices are used for repetitive motions that are not in needof any modulation.

Ingredient management and manipulation—refers to defining eachingredient in detail (including size, shape, weight, dimensions,characteristics and properties), one or more real-time adjustments inthe variables associated with the particular ingredient that may differfrom the previous stored ingredient details (such as the size of a fishfillet, the dimensions of an egg, etc.), and the process in executingthe different stages for the manipulation movements to an ingredient.

Kitchen Module (or Kitchen Volume)—a standardized full kitchen modulewith standardized sets of kitchen equipment, standardized sets ofkitchen tools, standardized sets of kitchen handles, and standardizedsets of kitchen containers, with predefined space and dimensions forstoring, accessing, and operating each kitchen element in thestandardized full kitchen module. One objective of a kitchen module isto predefine as much of the kitchen equipment, tools, handles,containers, etc. as possible so as to provide a relatively fixed kitchenplatform for the movements of robotic arms and hands. Both a chef in thechef kitchen studio and a person at home with a robotic kitchen (or aperson at a restaurant) uses the standardized kitchen module so as tomaximize the predictability of the kitchen hardware, while minimizingthe risks of differentiations, variations and deviations between thechef kitchen studio and a home robotic kitchen. Different embodiments ofthe kitchen module are possible, including a standalone kitchen moduleand an integrated kitchen module. The integrated kitchen module isfitted into a conventional kitchen area of a typical house. The kitchenmodule operates in at least two modes, a robotic mode and a normal(manual) mode.

Machine Learning—refers to the technology wherein a software componentor program improves its performance based on experience and feedback.One kind of machine learning is reinforcement learning, often used inrobotics, where desirable actions are rewarded and undesirable ones arepenalized. Another kind is case-based learning, where previoussolutions, e.g. sequences of actions by a human teacher or by the robotitself are remembered, together with any constraints or reasons for thesolutions, and then are applied or reused in new settings. There arealso additional kinds of machine learning, such as inductive andtransductive methods.

Mini-Manipulation—refers to a combination (or a sequence) of one or moresteps that accomplish a basic functional outcome with a threshold valueof the highest level of probability (examples of threshold value aswithin 0.1, 0.001, or 0.001 of the optimal value). Each step can be anaction primitive or another (smaller) mini-manipulation, similar to acomputer program comprised of basic coding steps and other computerprograms that may stand alone or serve as sub-routines. For instance, amini-manipulation can be grasping an egg, comprised of the motor actionsrequired for reaching out a robotic arm moving the robotic fingers intothe right configuration, and applying the correct delicate amount offorce for grasping—all primitive actions. Another mini-manipulation canbe breaking-an-egg-with-a-knife, including the graspingmini-manipulation, followed with one robotic hand, followed bygrasping-a-knife mini-manipulation with the other hand, followed by theprimitive action of striking the egg with the knife using apredetermined force.

Model Elements and Classification—refers to one or more software-basedcomputer program(s) capable of understanding elements in a scene asbeing items that are used or needed in different parts of a task; suchas a bowl for mixing and the need for a spoon to stir, etc. Multipleelements in a scene or a world-model may be classified into groupingsallowing for faster planning and task-execution.

Motion Primitives—refers to motion actions that define differentlevels/domains of detailed action steps, e.g. a high level motionprimitive would be to grab a cup, and a low level motion primitive wouldbe to rotate a wrist by five degrees.

Multimodal Sensing Unit—refers to a sensing unit comprised of multiplesensors capable of sensing and detection in multiple modes orelectromagnetic bands or spectra, particularly capable of capturingthree-dimensional position and/or motion information; theelectromagnetic spectrum can range from low to high frequencies and neednot be limited to that perceivable by a human being. Additional modesmight include, but are not limited to, other physical senses such astouch, smell, etc.

Number of Axes—three axes are required to reach any point in space. Tofully control the orientation of the end of the arm (i.e. the wrist),three additional rotational axes (yaw, pitch, and roll) are required.

Parameters—refers to variables that can take numerical values or rangesof numerical values. Three kinds of parameters are particularlyrelevant: parameters in the instructions to a robotic device (e.g. theforce or distance in an arm movement), user settable parameters (e.g.prefers meat well done vs. medium), and chef-defined parameters (e.g.set oven temperature to 350 F).

Parameter adjustment—refers to the process of changing the values ofparameters based on inputs. For instance changes in the parameters ofinstructions to the robotic device can be based on the properties (e.g.size, shape, orientation) of, but not limited to, the ingredients,position/orientation of kitchen tools, equipment, appliances, speed, andtime duration of a mini-manipulation.

Payload or carrying capacity—refers to how much weight a robotic arm cancarry and hold (or even accelerate) against the force of gravity, as afunction of its endpoint location.

Physical Reasoning—refers to a software-based computer program(s)capable of relying on geometrically-reasoned data and using physicalinformation (density, texture, typical geometry and shape) to assist aninference-engine (program) to better model the object and also predictits behavior in the real world, particularly when grasped and/ormanipulated/handled.

Raw Data—refers to all measured and inferred sensory-data andrepresentation information that is collected as part of the chef-studiorecipe-generation process while watching/monitoring a human chefpreparing a dish. Raw data can range from a simple data-point such asclock-time, to oven temperature (over time), camera-imagery,three-dimensional laser-generated scene representation data, toappliances/equipment used, tools employed, ingredients (type and amount)dispensed and when, etc. All the information the studio-kitchen collectsfrom its built-in sensors and stores in raw, time-stamped form isconsidered raw data. Raw data is then used by other software processesto generate an even higher level of understanding and recipe-processunderstanding, turning raw data into additional time-stampedprocessed/interpreted data.

Robotic Apparatus—refers the set of robotic sensors and effectors. Theeffectors comprise one or more robotic arms, and one or more robotichands for operation in the standardized robotic kitchen. The sensorscomprise cameras, range sensors, force sensors (haptic sensors) thattransmit their information to the processor or set of processors thatcontrol the effectors.

Recipe Cooking Process—refers to a robotic script containing abstractand detailed levels of instructions to a collection of programmable andhard automation devices, so as to allow computer-controllable devices toexecute a sequenced operation within its environment (e.g. a kitchenreplete with ingredients, tools, utensils and appliances).

Recipe Script—refers to a recipe script as a sequence in time containinga structure and a list of commands and execution primitives (simple tocomplex command software) that, when executed by the robotic kitchenelements (robot-arm, automated equipment, appliances, tools, etc.) in agiven sequence, should result in the proper replication and creation ofthe same dish as prepared by the human chef in the studio-kitchen. Sucha script is sequential in time and equivalent to the sequence employedby the human chef to create the dish, albeit in a representation that issuitable and understandable by the computer-controlled elements in therobotic kitchen.

Recipe Speed Execution—refers to managing a timeline in the execution ofrecipe steps in preparing a food dish by replicating a chef's movements,where the recipe steps include standardized food preparation operations(e.g., standardized cookware, standardized equipment, kitchenprocessors, etc.), mini-manipulations, and cooking of non-standardizedobjects.

Repeatability—refers to an acceptable preset margin in how accuratelythe robotic arms/hands can repeatedly return to a programmed position.If the technical specification in a control memory requires the robotichand to move to a certain X-Y-Z position and within +/−0.1 mm of thatposition, then the repeatability is measured for the robotic hands toreturn to within +/−0.1 mm of the taught and desired/commanded position.

Robotic Recipe Script—refers to a computer-generated sequence ofmachine-understandable instructions related to the proper sequence ofrobotically/hard-automation execution of steps to mirror the requiredcooking steps in a recipe to arrive at the same end-product as if cookedby a chef.

Robotic Costume—External instrumented device(s) or clothing, such asgloves, clothing with camera-trackable markers, jointed exoskeleton,etc., used in the chef studio to monitor and track the movements andactivities of the chef during all aspects of the recipe cookingprocess(es).

Scene Modeling—refers to a software-based computer program(s) capable ofviewing a scene in one or more cameras' fields of view, and beingcapable of detecting and identifying objects of importance to aparticular task. These objects may be pre-taught and/or be part of acomputer library with known physical attributes and usage-intent.

Smart Kitchen Cookware/Equipment—refers to an item of kitchen cookware(e.g., a pot or a pan) or an item of kitchen equipment (e.g., an oven, agrill, or a faucet) with one or more sensors that prepares a food dishbased on one or more graphical curves (e.g., a temperature curve, ahumidity curve, etc.).

Software Abstraction Food Engine—refers to a software engine that isdefined as a collection of software loops or programs, acting in concertto process input data and create a certain desirable set of output datato be used by other software engines or an end-user through some form oftextual or graphical output interface. An abstraction software engine isa software program(s) focused on taking a large and vast amount of inputdata from a known source in a particular domain (such asthree-dimensional range measurements that form a data-cloud ofthree-dimensional measurements as seen by one or more sensors), and thenprocessing the data to arrive at interpretations of the data in adifferent domain (such as detecting and recognizing a table-surface in adata-cloud based on data having the same vertical data value, etc.), inorder to identify, detect and classify data-readings as pertaining to anobject in three-dimensional space (such as a table-top, cooking pot,etc.). The process of abstraction is basically defined as taking a largedata set from one domain and inferring structure (such as geometry) in ahigher level of space (abstracting data points), and then abstractingthe inferences even further and identifying objects (pots, etc.) out ofthe abstracted data-sets to identify real-world elements in an image,which can then be used by other software engines to make additionaldecisions (handling/manipulation decisions for key objects, etc.). Asynonym for “software abstraction engine” in this application could bealso “software interpretation engine” or even “computer-softwareprocessing and interpretation algorithm”.

Task Reasoning—refers to a software-based computer program(s) capable ofanalyzing a task-description and breaking it down into a sequence ofmultiple machine-executable (robot or hard-automation systems) steps soas to achieve a particular end result defined in the task description.

Three-dimensional World Object Modeling and Understanding—refers to asoftware-based computer program(s) capable of using sensory data tocreate a time-varying three-dimensional model of all surfaces andvolumes so as to enable it to detect, identify and classify objectswithin the same and understand their usage and intent.

Torque vector—refers to the torsion force upon a robotic appendageincluding its direction and magnitude.

Volumetric Object Inference (Engine)—refers to a software-based computerprogram(s) capable of using geometric data and edge-information as wellas other sensory data (color, shape, texture, etc.) to allow foridentification of three-dimensionality of one or more objects to aid inthe object identification and classification process.

FIG. 1 is a system diagram illustrating an overall robotic foodpreparation kitchen 10 with robotics hardware 12 and robotics software14. The overall robotics food preparation kitchen 10 comprises roboticsfood preparation hardware 12 and robotics food preparation software 14that operate together to perform the robotics functions for foodpreparation. The robotic food preparation hardware 12 includes acomputer 16 that controls the various operations and movements of astandardized kitchen module 18 (which generally operate in aninstrumented environment with one or more sensors) multimodalthree-dimensional sensors 20, robotic arms 22, robotic hands 24 andcapturing gloves 26. The robotic food preparation software 14 operateswith the robotics food preparation hardware 12 to capture a chef'smovements in preparing a food dish and replicating the chef's movementsvia robotics arms and hands to obtain the same result or substantiallythe same result (e.g., taste the same, smell the same, etc.) of the fooddish that would taste the same or substantially the same as if the fooddish was prepared by a human chef.

The robotic food preparation software 14 includes the multimodalthree-dimensional sensors 20, a capturing module 28, a calibrationmodule 30, a conversion algorithm module 32, a replication module 34, aquality check module 36 with a three-dimensional vision system, a sameresult module 38, and a learning module 40. The capturing module 28captures the movements of the chef as the chef prepares a food dish. Thecalibration module 30 calibrates the robotic arms 22 and robotic hands24 before, during and after the cooking process. The conversionalgorithm module 32 is configured to convert the recorded data from achef's movements collected in the chef studio into recipe modified data(or transformed data) for use in a robotic kitchen where robotic handsreplicate the food preparation of the chef's dish. The replicationmodule 34 is configured to replicate the chef's movements in a robotickitchen. The quality check module 36 is configured to perform qualitycheck functions of a food dish prepared by the robotic kitchen during,prior to, or after the food preparation process. The same result module38 is configured to determine whether the food dish prepared by a pairof robotic arms and hands in the robotic kitchen would taste the same orsubstantially the same as if prepared by the chef. The learning module40 is configured to provide learning capabilities to the computer 16that operates the robotic arms and hands.

FIG. 2 is a system diagram illustrating a first embodiment of a foodrobot cooking system that includes a chef studio system and a householdrobotic kitchen system for preparing a dish by replicating a chef'srecipe process and movements. The robotic kitchen cooking system 42comprises a chef kitchen 44 (also referred to as “chef studio-kitchen”)which transfers one or more software recorded recipe files 46 to arobotic kitchen 48 (also referred to as “household robotic kitchen”). Inone embodiment, both the chef kitchen 44 and the robotic kitchen 48 usethe same standardized robotic kitchen module 50 (also referred as“robotic kitchen module”, “robotic kitchen volume”, or “kitchen module”,or “kitchen volume”) to maximize the precise replication of preparing afood dish, which reduces the variables that may contribute to deviationsbetween the food dish prepared at the chef kitchen 44 and the oneprepared by the robotic kitchen 46. A chef 52 wears robotic gloves or acostume with external sensory devices for capturing and recording thechef's cooking movements. The standardized robotic kitchen 50 comprisesa computer 16 for controlling various computing functions, where thecomputer 16 includes a memory 52 for storing one or more software recipefiles from the sensors of the gloves or costumes 54 for capturing achef's movements, and a robotic cooking engine (software) 56. Therobotic cooking engine 56 includes a movement analysis and recipeabstraction and sequencing module 58. The robotic kitchen 48 typicallyoperates with a pair of robotic arms and hands, with an optional user 60to turn on or program the robotic kitchen 46. The computer 16 in therobotic kitchen 48 includes a hard automation module 62 for operatingrobotic arms and hands, and a recipe replication module 64 forreplicating a chef's movements from a software recipe (ingredients,sequence, process, etc.) file.

The standardized robotic kitchen 50 is designed for detecting, recordingand emulating a chef's cooking movements, controlling significantparameters such as temperature over time, and process execution atrobotic kitchen stations with designated appliances, equipment andtools. The chef kitchen 44 provides a computing kitchen environment 16with gloves with sensors or a costume with sensors for recording andcapturing a chef's 50 movements in the food preparation for a specificrecipe. Upon recording the movements and recipe process of the chef 49for a particular dish into a software recipe file in memory 52, thesoftware recipe file is transferred from the chef kitchen 44 to therobotic kitchen 48 via a communication network 46, including a wirelessnetwork and/or a wired network connected to the Internet, so that theuser (optional) 60 can purchase one or more software recipe files or theuser can be subscribed to the chef kitchen 44 as a member that receivesnew software recipe files or periodic updates of existing softwarerecipe files. The household robotic kitchen system 48 serves as arobotic computing kitchen environment at residential homes, restaurants,and other places in which the kitchen is built for the user 60 toprepare food. The household robotic kitchen system 48 includes therobotic cooking engine 56 with one or more robotic arms andhard-automation devices for replicating the chef's cooking actions,processes and movements based on a received software recipe file fromthe chef studio system 44.

The chef studio 44 and the robotic kitchen 48 represent an intricatelylinked teach-playback system, which has multiple levels of fidelity ofexecution. While the chef studio 44 generates a high-fidelity processmodel of how to prepare a professionally cooked dish, the robotickitchen 48 is the execution/replication engine/process for therecipe-script created through the chef working in the chef studio.Standardization of a robotic kitchen module is a means to increaseperformance fidelity and success/guarantee.

The varying levels of fidelity for recipe-execution depend on thecorrelation of sensors and equipment (besides of course the ingredients)between those in the chef studio 44 and that in the robotic kitchen 48.Fidelity can be defined as a dish tasting identical to that prepared bya human chef (indistinguishably so) at one of the (perfectreplication/execution) ends of the spectrum, while at the opposite endthe dish could have one or more substantial or fatal flaws withimplications to quality (overcooked meat or pasta), taste (burntelements), edibility (incorrect consistency) or even health-implications(undercooked meat such as chicken/pork with salmonella exposure, etc.).

A robotic kitchen that has identical hardware and sensors and actuationsystems that can replicate the movements and processes akin to those bythe chef that were recorded during the chef-studio cooking process ismore likely to result in a higher fidelity outcome. The implication hereis that the setups need to be identical, which has a cost and volumeimplication. The robotic kitchen 48 can however still be implementedusing more standardized non-computer-controlled or computer-monitoredelements (pots with sensors, networked appliances such as ovens, etc.),requiring more sensor-based understanding to allow for more complexexecution monitoring. Since uncertainty has now increased as to keyelements (correct amount of ingredients, cooking temperatures, etc.) andprocesses (use of stirrer/masher in case a blender is not available in arobotic home kitchen), the guarantees of having an identical outcome tothat from the chef will undoubtedly be lower.

An emphasis in the present disclosure is that the notion of a chefstudio 44 coupled with a robotic kitchen is a generic concept. The levelof the robotic kitchen 48 is variable all the way from a home-kitchenoutfitted with a set of arms and environmental sensors, all the way toan identical replica of the studio-kitchen, where a set of arms andarticulated motions, tools and appliances and ingredient-supply canreplicate the chef's recipe in an almost identical fashion. The onlyvariable to contend with will be the quality-degree of the end-result ordish in terms of quality, looks, taste, edibility and health.

A potential method to mathematically describe this correlation betweenthe recipe-outcome and the input variables in the robotic kitchen canbest be described by the function below:F _(recipe-outcome) =F _(studio)(I,E,P,M,V)+F _(RobKit)(E _(f) ,I,R _(e),P _(mf))

-   -   where F_(studio)=Recipe Script Fidelity of Chef-Studio        -   F_(RobKit)=Recipe Script Execution by Robotic Kitchen        -   I=Ingredients        -   E=Equipment        -   P=Processes        -   M=Methods        -   V=Variables (Temperature, Time, Pressure, etc.)        -   E_(f)=Equipment Fidelity        -   R_(e)=Replication Fidelity        -   P_(mf)=Process Monitoring Fidelity

The above equation relates the degree to which the outcome of arobotically-prepared recipe matches that a human chef would prepare andserve (F_(recipe-outcome)) to the level that the recipe was properlycaptured and represented by the chef studio 44 (F_(studio)) based on theingredients (I) used, the equipment (E) available to execute the chef'sprocesses (P) and methods (M) by properly capturing all the keyvariables (V) during the cooking process; and how the robotic kitchen isable to represent the replication/execution process of the roboticrecipe script by a function (F_(RobKit)) that is primarily driven by theuse of the proper ingredients (I), the level of equipment fidelity(E_(f)) in the robotic kitchen compared to that in the chef studio, thelevel to which the recipe-script can be replicated (R_(e)) in therobotic kitchen, and to what extent there is an ability and need tomonitor and execute corrective actions to achieve the highest processmonitoring fidelity (P_(mf)) possible.

The functions (F_(studio)) and (F_(RobKit)) can be any combination oflinear or non-linear functional formulas with constants, variables andany form of algorithmic relationships. An example for such algebraicrepresentations for both functions could be.F _(studio) =I(fct. sin(Temp))+E(fct. Cooptop1*5)+P(fct.Circle(spoon)+V(fct. 0.5*time)

Delineating that the fidelity of the preparation process is related tothe temperature of the ingredient which varies over time in therefrigerator as a sinusoidal function, the speed with which aningredient can be heated on the cooktop on specific station at aparticular multiplicative rate, and related to how well a spoon can bemoved in a circular path of a certain amplitude and period, and that theprocess needs to be carried out at no less than ½ the speed of the humanchef for the fidelity of the preparation process to be maintained.F _(RobKit) =E _(f),(Cooktop2,Size)+I(1.25*Size+Linear(Temp))+R_(e)(Motion-Profile)+P _(mf)(Sensor-Suite Correspondence)

Delineating that the fidelity of the replication process in the robotickitchen is related to the appliance type and layout for a particularcooking-area and the size of the heating-element, the size andtemperature profile of the ingredient being seared and cooked (thickersteak requiring more cooking time), while also preserving themotion-profile of any stirring and bathing motions of a particular steplike searing or mousse-beating, and whether the correspondence betweensensors in the robotic kitchen and the chef-studio is sufficiently highto trust the monitored sensor data to be accurate and detailed enough toprovide a proper monitoring fidelity of the cooking process in therobotic kitchen during all steps in a recipe.

The outcome of a recipe is not only a function of what fidelity thehuman chef's cooking steps/methods/process/skills were captured with bythe chef studio, but also with what fidelity these can be executed bythe robotic kitchen, where each of them has key elements that impacttheir respective subsystem performance.

FIG. 3 is a system diagram illustrating one embodiment of thestandardized robotic kitchen 50 for food preparation by recording achef's movement in preparing a food dish and replicating the food dishby robotic arms and hands. In this context, the term “standardized” (or“standard”) means that the specifications of the components or featuresare presets, as will be explained below. The computer 16 iscommunicatively coupled to multiple kitchen elements in the standardizedrobotic kitchen 50, including a three-dimensional vision sensor 66, aretractable safety screen (e.g., glass, plastic, or other types ofprotective material) 68, robotic arms 70, robotic hands 72, standardizedcooking appliances/equipment 74, standardized cookware with sensors 76,standardized cookware 78, standardized handles and utensils 80,standardized hard automation dispenser(s) 82 (also referred to as“robotic hard automation module(s)”), a standardized kitchen processor84, standardized containers 86, and a standardized food storage in arefrigerator 88.

The standardized hard automation dispenser(s) 82 is a device or a seriesof devices that is/are programmable and/or controllable via the cookingcomputer 16 to feed or provide pre-packaged (known) amounts or dedicatedfeeds of key materials for the cooking process, such as spices (salt,pepper, etc.), liquids (water, oil, etc.) or other dry materials (flour,sugar, etc.). The standardized hard automation dispensers 82 may belocated at a specific station or be able to be robotically accessed andtriggered to dispense according to the recipe sequence. In otherembodiments, a robotic hard automation module may be combined orsequenced in series or parallel with other such modules or robotic armsor cooking utensils. In this embodiment, the standardized robotickitchen 50 includes robotic arms 70 and robotic hands 72 and robotichands as controlled by the robotic food preparation engine 56 inaccordance with a software recipe file stored in the memory 52 forreplicating a chef's precise movements in preparing a dish to producethe same tasting dish as if the chef had prepared it himself or herself.The three-dimensional vision sensors 66 provide capability to enablethree-dimensional modeling of objects, providing a visualthree-dimensional model of the kitchen activities, and scanning thekitchen volume to assess the dimensions and objects within thestandardized robotic kitchen 50. The retractable safety glass 68comprises a transparent material on the robotic kitchen 50, which whenin an ON state extends the safety glass around the robotic kitchen toprotect surrounding human beings from the movements of robotic arms 70and hands 72, hot water and other liquids, steam, fire and other dangersinfluents. The robotic food preparation engine 56 is communicativelycoupled to an electronic memory 52 for retrieving a software recipe filepreviously sent from the chef studio system 44 for which the roboticfood preparation engine 56 is configured to execute processes inpreparing and replicating the cooking method and processes of a chef asindicated in the software recipe file. The combination of robotic arms70 and robotic hands 72 serves to replicate the precise movements of thechef in preparing a dish so that the resulting food dish will tasteidentical (or substantially identical) to the same food dish prepared bythe chef. The standardized cooking equipment 74 includes an assortmentof cooking appliances 46 that are incorporated as part of the robotickitchen 50, including, but not limited to, a stove/induction/cooktop(electric cooktop, gas cooktop, induction cooktop), an oven, a grill, acooking steamer, and a microwave oven. The standardized cookware andsensors 76 are used as embodiments for the recording of food preparationsteps based on the sensors on the cookware and cooking a food dish basedon the cookware with sensors, which include a pot with sensors, a panwith sensors, an oven with sensors, and a charcoal grill with sensors.The standardized cookware 78 includes frying pans, sauté pans, grillpans, multi-pots, roasters, woks, and braisers. The robotic arms 70 andthe robotic hands 72 operate the standardized handles and utensils 80 inthe cooking process. In one embodiment, one of the robotic hands 72 isfitted with a standardized handle, which is attached to a fork head, aknife head, and a spoon head for selection as required. The standardizedhard automation dispensers 82 are incorporated into the robotic kitchen50 to provide for expedient (via both robot arms 70 and human use) keyand common/repetitive ingredients that are easily measured/dosed out orpre-packaged. The standardized containers 86 are storage locations thatstore food at room temperature. The standardized refrigerator containers88 refer to, but are not limited to, a refrigerator with identifiedcontainers for storing fish, meat, vegetables, fruit, milk, and otherperishable items. The containers in the standardized containers 86 orstandardized storages 88 can be coded with container identifiers fromwhich the robotic food preparation engine 56 is able to ascertain thetype of food in a container based on the container identifier. Thestandardized containers 86 provide storage space for non-perishable fooditems such as salt, pepper, sugar, oil, and other spices. Standardizedcookware with sensors 76 and the cookware 78 may be stored on a shelf ora cabinet for use by the robotic arms 70 for selecting a cooking tool toprepare a dish. Typically, the raw fish, the raw meat, and vegetablesare pre-cut and stored in the identified standardized storages 88. Thekitchen countertop 90 provides a platform for the robotic arms 70 tohandle the meat or vegetables as needed, which may or may not includecutting or chopping actions. The kitchen faucet 92 provides a kitchensink space for washing or cleaning food in preparation for a dish. Whenthe robotic arms 70 have completed the recipe process to prepare a dishand the dish is ready for serving, the dish is placed on a servingcounter 90, which further allows for the dining environment to beenhanced by adjusting the ambient setting with the robotic arms 70, suchas placement of utensils, wine glasses, and a chosen wine compatiblewith the meal. One embodiment of the equipment in the standardizedrobotic kitchen module 50 is a professional series as to increase theuniversal appeal to prepare various types of dishes.

The standardized robotic kitchen module 50 has as one objective thestandardization of the kitchen module 50 and various components with thekitchen module itself, to ensure consistency in both the chef kitchen 44and the robotic kitchen 48 to maximize the preciseness of recipereplication while minimizing the risks of deviations from precisereplication of a recipe dish between the chef kitchen 44 and the robotickitchen 48. One main purpose of having the standardization of thekitchen module 50 is to obtain the same result of the cooking process(or the same dish) between a first food dish prepared by the chef and asubsequent replication of the same recipe process via the robotickitchen. Conceiving a standardized platform in the standardized robotickitchen module 50 between the chef kitchen 44 and the robotic kitchen 48has several key considerations: same timeline, same program or mode, andquality check. The same timeline in the standardized robotic kitchen 50where the chef prepares a food dish at the chef kitchen 44 and thereplication process by the robotic hands in the robotic kitchen 48refers to the same sequence of manipulations, the same initial andending time of each manipulation, and the same speed of moving an objectbetween handling operations. The same program or mode in thestandardized robotic kitchen 50 refers to the use and operation ofstandardized equipment during each manipulation recording and executionstep. The quality check refers to three-dimensional vision sensors inthe standardized robotic kitchen 50 which monitor and adjust in realtime each manipulation action during the food preparation process tocorrect any deviation and avoid a flawed result. The adoption of thestandardized robotic kitchen module 50 reduces and minimizes the risksof not obtaining the same result between the chef's prepared food dishand the food dish prepared by the robotic kitchen using robotic arms andhands. Without the standardization of a robotic kitchen module and thecomponents within the robotic kitchen module, the increased variationsbetween the chef kitchen 44 and the robotic kitchen 48 increase therisks of not being able to obtain the same result between the chef'sprepared food dish and the food dish prepared by the robotic kitchenbecause more elaborate and complex adjustment algorithms will berequired with different kitchen modules, different kitchen equipment,different kitchenware, different kitchen tools, and differentingredients between the chef kitchen 44 and the robotic kitchen 48.

The standardized robotic kitchen module 50 includes standardization ofmany aspects. First, the standardized robotic kitchen module 50 includesstandardized positions and orientations (in the XYZ coordinate plane) ofany type of kitchenware, kitchen containers, kitchen tools and kitchenequipment (with standardized fixed holes in the kitchen module anddevice positions). Secondly, the standardized robotic kitchen module 50includes a standardized cooking volume dimension and architecture.Thirdly, the standardized robotic kitchen module 50 includesstandardized equipment sets, such as an oven, a stove, a dish washer, afaucet, etc. Fourth, the standardized robotic kitchen module 50 includesstandardized kitchenware, standardized cooking tools, standardizedcooking devices, standardized containers, and standardized food storagein a refrigerator, in terms of shape, dimension, structure, material,capabilities, etc. Fifth, in one embodiment, the standardized robotickitchen module 50 includes a standardized universal handle for handlingany kitchenware, tools, instruments, containers, and equipment, whichenable a robotic hand to hold the standardized universal handle in onlyone correct position, while avoiding any improper grasps or incorrectorientations. Sixth, the standardized robotic kitchen module 50 includesstandardized robotic arms and hands with a library of manipulations.Seventh, the standardized robotic kitchen module 50 includes astandardized kitchen processor for standardized ingredientmanipulations. Eighth, the standardized robotic kitchen module 50includes standardized three-dimensional vision devices for creatingdynamic three-dimensional vision data, as well as other possiblestandard sensors, for recipe recording, execution tracking, and qualitycheck functions. Ninth, the standardized robotic kitchen module 50includes standardized types, standardized volumes, standardized sizes,and standardized weights for each ingredient during a particular recipeexecution.

FIG. 4 is a system diagram illustrating one embodiment of the roboticcooking engine 56 (also referred to as “robotic food preparationengine”) for use with the computer 16 in the chef studio system 44 andthe household robotic kitchen system 48. Other embodiments may havemodifications, additions, or variations of the modules in the roboticcooking engine 16 in the chef kitchen 44 and robotic kitchen 48. Therobotic cooking engine 56 includes an input module 50, a calibrationmodule 94, a quality check module 96, a chef movement recording module98, a cookware sensor data recording module 100, a memory module 102 forstoring software recipe files, a recipe abstraction module 104 usingrecorded sensor data to generate machine-module specific sequencedoperation profiles, a chef movements replication software module 106, acookware sensory replication module 108 using one or more sensorycurves, a robotic cooking module 110 (computer control to operatestandardized operations, mini-manipulations, and non-standardizedobjects), a real-time adjustment module 112, a learning module 114, amini-manipulation library database module 116, a standardized kitchenoperation library database module 117, and an output module 118, towhich these modules are communicatively coupled via a bus 120.

The input module 50 is configured to receive any type of inputinformation such as software recipe files sent from another computingdevice. The calibration module 94 is configured to calibrate itself withthe robotic arms 70, the robotic hands 72, and other kitchenware andequipment components within the standardized robotic kitchen module 50.The quality check module 96 is configured to determine the quality andfreshness of raw meat, raw vegetables, milk-associated ingredients andother raw foods at the time that the raw food is retrieved for cooking,as well as checking the quality of raw foods when receiving the foodinto the standardized food storage 88. The quality check module 96 canalso be configured to conduct quality testing of an object based onsenses, such as the smell of the food, the color of the food, the tasteof the food, and the image or appearance of the food. The chef movementsrecording module 98 is configured to record the sequence and the precisemovements of the chef when the chef prepares a food dish. The cookwaresensor data recording module 100 is configured to record sensory datafrom cookware equipped with sensors (such as a pan with sensors, a grillwith sensors, or an oven with sensors) placed in different zones withinthe cookware, thereby producing one or more sensory curves. The resultis the generation of a sensory curve, such as temperature curve (and/orhumidity), that reflects the temperature fluctuation of cookingappliances over time for a particular dish. The memory module 102 isconfigured as a storage location for storing software recipe files, foreither replication of chef recipe movements or other types of softwarerecipe files including sensory data curves. The recipe abstractionmodule 104 is configured to use recorded sensor data to generatemachine-module specific sequenced operation profiles. The chef movementsreplication module 106 is configured to replicate the chef's precisemovements in preparing a dish based on the stored software recipe filein the memory 52. The cookware sensory replication module 108 isconfigured to replicate the preparation of a food dish by following thecharacteristics of one or more previously recorded sensory curves whichwas generated when the chef 49 prepared a dish by using the standardizedcookware with sensors 76. The robotic cooking module 110 is configuredto control and operate standardized kitchen operations,mini-manipulations, non-standardized objects, and the various kitchentools and equipment in the standardized robotic kitchen 50. The realtime adjustment module 112 is configured to provide real-timeadjustments to the variables associated with a particular kitchenoperation or a mini operation so as to produce a resulting process thatis a precise replication of the chef movement or a precise replicationof the sensory curve. The learning module 114 is configured to providelearning capabilities to the robotic cooking engine 56 to optimize theprecise replication in preparing a food dish by robotic arms 70 and therobotic hands 72, as if the food dish was prepared by a chef, using amethod such as case-based (robotic) learning. The mini-manipulationlibrary database module 116 is configured to store

a first database library of mini-manipulations. The standardized kitchenoperation library database module 117 is configured to store a seconddatabase library of standardized kitchenware and how to operate thisstandardized kitchenware. The output module 118 is configured to sendoutput computer files or control signals external to the robotic cookingengine.

FIG. 5A is a block diagram illustrating a chef studio recipe-creationprocess, showcasing several main functional blocks supporting the use ofexpanded multimodal sensing to create a recipe instruction-script for arobotic kitchen. Sensor-data from a multitude of sensors, such as (butnot limited to) smell 124, video cameras 126, infrared scanners andrangefinders 128, stereo (or even trinocular) cameras 130, haptic gloves132, articulated laser-scanners 134, virtual-world goggles 136,microphones 138 or an exoskeletal motion suit 140, human voice 142,touch-sensors 144 and even other forms of user input 146, are used tocollect data through a sensor interface module 148. The data is acquiredand filtered 150, including possible human user input (e.g., chef;touch-screen and voice input) 146, after which a multitude of (parallel)software processes utilize the temporal and spatial data to generate thedata that is used to populate the machine-specific recipe-creationprocess. Sensors may not be limited to capturing human position and/ormotion but may also capture position, orientation and/or motion of otherobjects in the standardized robotic kitchen 50.

These individual software modules generate such information (but are notthereby limited to only these modules) as (i) chef-location andcooking-station ID via a location and configuration module 152, (ii)configuration of arms (via torso), (iii) tools handled and when and how,(iv) utensils used and locations on the station through the hardware andvariable abstraction module 154, (v) processes executed with them and(vi) variables (temperature, lid y/n, stirring, etc.) in need ofmonitoring through the process module B156, (vii) temporal(start/finish, type) distribution and (viii) types of processes (stir,fold, etc.) being applied, and (ix) ingredients added (type, amount,state of prep, etc.), through the cooking sequence and processabstraction module 158.

All this information is then used to create a machine-specific (not justfor the robotic-arms, but also ingredient dispensers, tools andutensils, etc.) set of recipe instructions through the stand-alonemodule 160, which are organized as a script of sequential/paralleloverlapping tasks to be executed and monitored. This recipe-script isstored (162) alongside the entire raw data set (164) in the data storagemodule 166 and is made accessible to either a remote robotic cookingstation through the robotic kitchen interface module 168 or a human user170 via a graphical user interface (GUI) 172.

FIG. 5B is a block diagram illustrating one embodiment of thestandardized chef studio 44 and robotic kitchen 50 with teach/playbackprocess 176. The teach/playback process 176 describes the steps ofcapturing a chef's recipe-implementation processes/methods/skills 49 inthe chef studio 44 where he/she carries out the recipe execution 180,using a set of chef-studio standardized equipment 74 and recipe-requiredingredients 178 to create a dish while being logged and monitored 182.The raw sensor data is logged (for playback) in 182 and also processedto generate information at different abstraction levels (tools/equipmentused, techniques employed, times/temperatures started/ended, etc.), andthen used to create a recipe-script 184 for execution by the robotickitchen 48.

The robotic kitchen 48 engages in a recipe replication process 106,whose profile depends on whether the kitchen is of a standardized ornon-standardized type, which is checked by a process 186.

The robotic kitchen execution is dependent on the type of kitchenavailable to the user. If the robotic kitchen uses the same/identical(at least functionally) equipment as used in the in the chef studio, therecipe replication process is primarily one of using the raw data andplaying it back as part of the recipe-script execution process. Shouldthe kitchen however differ from the (ideal) standardized kitchen, theexecution engine(s) will have to rely on the abstracted data to generatekitchen-specific execution sequences to try to achieve a similarstep-by-step result.

Since the cooking process is continually monitored by all sensor unitsin the robotic kitchen via a monitoring process 194, regardless ofwhether the known studio equipment 196 or the mixed/atypical non-chefstudio equipment 198 is being used, the system is able to makemodifications as needed depending on a recipe progress check 200. In oneembodiment of the standardized kitchen, raw data is typically playedback through an execution module 188 using chef-studio type equipment,and the only adjustments that are expected are adaptations 202 in theexecution of the script (repeat a certain step, go back to a certainstep, slow down the execution, etc.) as there is a one-to-onecorrespondence between taught and played-back data-sets. However, in thecase of the non-standardized kitchen, the chances are very high that thesystem will have to modify and adapt the actual recipe itself and itsexecution via a recipe script modification module 204, to suit theavailable tools/appliances 192 which differ from those in the chefstudio 44 or the measured deviations from the recipe script (meatcooking too slowly, hot-spots in pot burning the roux, etc.). Overallrecipe-script progress is monitored using a similar process 206, whichdiffers depending on whether chef-studio equipment 208 or mixed/atypicalkitchen equipment 210 is being used.

A non-standardized kitchen is less likely to result in a close-to-humanchef cooked dish, as compared to using a standardized robotic kitchenthat has equipment and capabilities reflective of those used in thestudio-kitchen. The ultimate subjective decision is of course that ofthe human (or chef) tasting, which is a quality evaluation 212, yieldingto a (subjective) quality decision 214.

FIG. 5C is a block diagram illustrating one embodiment 216 of a recipescript generation and abstraction engine that pertains to the structureand flow of the recipe-script generation process as part of thechef-studio recipe walk-through by a human chef. The first step is forall available data measurable in the chef studio 44, whether it beergonomic data from the chef (arms/hands positions and velocities,haptic finger data, etc.), status of the kitchen appliances (ovens,fridges, dispensers, etc.), specific variables (cooktop temperature,ingredient temperature, etc.), appliance or tools being used (pots/pans,spatulas, etc.), or two-dimensional and three-dimensional data collectedby multi-spectrum sensory equipment (including cameras, lasers,structured light systems, etc.), to be input and filtered by the centralcomputer system and also time-stamped by a main process 218.

A data process-mapping algorithm 220 uses the simpler (typicallysingle-unit) variables to determine where the process action is takingplace (cooktop and/or oven, fridge, etc.) and assigns a usage tag to anyitem/appliance/equipment being used whether intermittently orcontinuously. It associates a cooking step (baking, grilling,ingredient-addition, etc.) to a specific time-period and tracks when,where and which and how much of what ingredient was added. This(time-stamped) information dataset is then made available for thedata-melding process during the recipe-script generation process 222.

The data extraction and mapping process 224 is primarily focused ontaking two-dimensional information (such as from monocular/single-lensedcameras) and extracting key information from the same. In order toextract the important and more abstracted descriptive information fromeach successive image, several algorithmic processes have to be appliedto this dataset. Such processing steps can include (but are not limitedto) edge-detection, color and texture-mapping, and then using thedomain-knowledge in the image, coupled with object-matching information(type and size) extracted from the data reduction and abstractionprocess 226, to allow for the identification and location of the object(whether an item of equipment or ingredient, etc.), again extracted fromthe data reduction and abstraction process 226, allowing one toassociate the state (and all associated variables describing the same)and items in an image with a particular process-step (frying, boiling,cutting, etc.). Once this data has been extracted and associated with aparticular image at a particular point in time, it can be passed to therecipe-script generation process 222 to formulate the sequence and stepswithin a recipe.

The data-reduction and abstraction engine (set of software routines) 226is intended to reduce the larger three-dimensional data sets and extractfrom them key geometric and associative information. A first step is toextract from the large three-dimensional data point-cloud only thespecific workspace area of importance to the recipe at that particularpoint in time. Once the data-set has been trimmed, key geometricfeatures will be identified by a process known as template matching;this allows for the identification of such items as horizontaltable-tops, cylindrical pots and pans, arm and hand locations, etc. Oncetypical known (template) geometric entities are determined in a data-seta process of object identification and matching proceeds todifferentiate all items (pot vs. pan, etc.) and associates the properdimensionality (size of pot or pan, etc.) and orientation of the same,and places them within the three-dimensional world model being assembledby the computer. All this abstracted/extracted information is then alsoshared with the data-extraction and mapping engine 224, prior to allbeing fed to the recipe-script generation engine 222.

The recipe-script generation engine process 222 is responsible formelding (blending/combining) all the available data and sets into astructured and sequential cooking script with clear process-identifiers(prepping, blanching, frying, washing, plating, etc.) andprocess-specific steps within each, which can then be translated intorobotic-kitchen machine-executable command-scripts that are synchronizedbased on process-completion and overall cooking time and cookingprogress. Data melding will at least involve, but will not solely belimited to, the ability to take each (cooking) process step andpopulating the sequence of steps to be executed with the properlyassociated elements (ingredients, equipment, etc.), methods andprocesses to be used during the process steps, and the associated keycontrol—(set oven/cooktop temperatures/settings) andmonitoring-variables (water or meat temperature, etc.) to be maintainedand checked to verify proper progress and execution. The melded data isthen combined into a structured sequential cooking script that willresemble a set of minimally descriptive steps (akin to a recipe in amagazine) but with a much larger set of variables associated with eachelement (equipment, ingredient, process, method, variable, etc.) of thecooking process at any one point in the procedure. The final step is totake this sequential cooking script and transform it into an identicallystructured sequential script that is translatable by a set ofmachines/robot/equipment within a robotic kitchen 48. It is this scriptthe robotic kitchen 48 uses to execute the automated recipe executionand monitoring steps.

All raw (unprocessed) and processed data as well as the associatedscripts (both structure sequential cooking-sequence script and themachine-executable cooking-sequence script) are stored in the data andprofile storage unit/process 228 and time-stamped. It is from thisdatabase that the user, by way of a GUI, can select and cause therobotic kitchen to execute a desired recipe through the automatedexecution and monitoring engine 230, which is continually monitored byits own internal automated cooking process, with necessary adaptationsand modifications to the script generated by the same and implemented bythe robotic-kitchen elements, in order to arrive at a completely platedand served dish.

FIG. 5D is a block diagram illustrating software elements forobject-manipulation in the standardized robotic kitchen, which shows thestructure and flow 250 of the object-manipulation portion of the robotickitchen execution of a robotic script, using the notion ofmotion-replication coupled-with/aided-by mini-manipulation steps. Inorder for automated robotic-arm/-hand-based cooking to be viable, it isinsufficient to simply monitor every single joint in the arm andhands/fingers. In many cases just the position and orientation of thehand/wrist are known (and able to be replicated), but then manipulatingan object (identifying location, orientation, pose, grab-location,grabbing-strategy and task-execution) requires that local-sensing andlearned behaviors and strategies for the hand and fingers be used tocomplete the grabbing/manipulating task successfully. Thesemotion-profiles (sensor-based/-driven) behaviors and sequences arestored within the mini hand-manipulation library software repository inthe robotic-kitchen system. The human chef could be wearing completearm-exoskeleton or an instrumented/target-fitted motion-vest allowingthe computer via built-in sensors or though camera-tracking to determinethe exact 3D position of the hands and wrists at all times. Even if theten fingers on both hands had all their joints instrumented (more than30 DoFs [Degrees of Freedom] for both hands and very awkward to wear anduse, and thus unlikely to be used), a simple motion-based playback ofall joint positions would not guarantee successful (interactive) objectmanipulation.

The mini-manipulation library is a command-software repository, wheremotion behaviors and processes are stored based on an off-line learningprocess, where the arm/wrist/finger motions and sequences tosuccessfully complete a particular abstract task (grab the knife andthen slice; grab the spoon and then stir; grab the pot with one hand andthen use other hand to grab spatula and get under meat and flip itinside the pan; etc.). This repository has been built up to contain thelearned sequences of successful sensor-driven motion-profiles andsequenced behaviors for the hand/wrist (and sometimes also arm-positioncorrections), to ensure successful completions of object (appliance,equipment, tools) and ingredient manipulation tasks that are describedin a more abstract language, such as “grab the knife and slice thevegetable”, “crack the egg into the bowl”, “flip the meat over in thepan”, etc. The learning process is iterative and is based on multipletrials of a chef-taught motion-profile from the chef studio, which isthen executed and iteratively modified by the offline learning algorithmmodule, until an acceptable execution-sequence can be shown to have beenachieved. The mini-manipulation library (command software repository) isintended to have been populated (a-priori and offline) with all thenecessary elements to allow the robotic-kitchen system to successfullyinteract with all equipment (appliances, tools, etc.) and mainingredients that require processing (steps beyond just dispensing)during the cooking process. While the human chef wore gloves withembedded haptic sensors (proximity, touch, contact-location/-force) forthe fingers and palm, the robotic hands are outfitted with similarsensor-types in locations to allow their data to be used to create,modify and adapt motion-profiles to successfully execute desiredmotion-profiles and handling-commands.

The object-manipulation portion of the robotic-kitchen cooking process(robotic recipe-script execution software module for the interactivemanipulation and handling of objects in the kitchen environment) 252 isfurther elaborated below. Using the robotic recipe-script database 254(which contains data in raw, abstracted cooking-sequence andmachine-executable script forms), the recipe script executor module 256steps through a specific recipe execution-step. The configurationplayback module 258 selects and passes configuration commands through tothe robot arm system (torso, arm, wrist and hands) controller 270, whichthen controls the physical system to emulate the required configuration(joint-positions/-velocities/-torques, etc.) values.

The notion of being able to faithfully carry out proper environmentinteraction manipulation and handling tasks is made possible through areal-time process-verification by way of (i) 3D world modeling as wellas (ii) mini-manipulation. Both the verification and manipulation stepsare carried out through the addition of the robot wrist and handconfiguration modifier 260. This software module uses data from the 3Dworld configuration modeler 262, which creates a new 3D world model atevery sampling step from sensory data supplied by the multimodalsensor(s) unit(s), in order to ascertain that the configuration of therobotic kitchen systems and process matches that required by the recipescript (database); if not, it enacts modifications to the commandedsystem-configuration values to ensure the task is completedsuccessfully. Furthermore, the robot wrist and hand configurationmodifier 260 also uses configuration-modifying input commands from themini-manipulation motion profile executor 264. The hand/wrist (andpotentially also arm) configuration modification data fed to theconfiguration modifier 260 are based on the mini-manipulation motionprofile executor 264 knowing what the desired configuration playbackshould be from 258, but then modifying it based on its 3D object modellibrary 266 and the a-priori learned (and stored) data from theconfiguration and sequencing library 268 (which was built based onmultiple iterative learning steps for all main object handling andprocessing steps).

While the configuration modifier 260 continually feeds modifiedcommanded configuration data to the robot arm system controller 270, itrelies on the handling/manipulation verification software module 272 toverify not only that the operation is proceeding properly but alsowhether continued manipulation/handling is necessary. In the case of thelatter (answer ‘N’ to the decision), the configuration modifier 260re-requests configuration-modification (for the wrist, hands/fingers andpotentially the arm and possibly even torso) updates from both the worldmodeler 262 and the mini-manipulation profile executor 264. The goal issimply to verify that a successful manipulation/handling step orsequence has been successfully completed. The handling/manipulationverification software module 272 carries out this check by using theknowledge of the recipe script database F2 and the 3D worldconfiguration modeler 262 to verify the appropriate progress in thecooking step currently being commanded by the recipe script executor256. Once progress has been deemed successful, the recipe script indexincrement process 274 notifies the recipe script executor 256 to proceedto the next step in the recipe-script execution.

FIG. 6 is a block diagram illustrating a multimodal sensing and softwareengine architecture 300 in accordance with the present invention. One ofthe main autonomous cooking features allowing for planning, executionand monitoring of a robotic cooking script requires the use ofmultimodal sensory input 302 that is used by multiple software modulesto generate data needed to (i) understand the world, (ii) model thescene and materials, (iii) plan the next steps in the robotic cookingsequence, (iv) execute the generated plan and (v) monitor the executionto verify proper operations—all of these steps occurring in acontinuous/repetitive closed loop fashion.

The multimodal sensor-unit(s) 302, comprising, but not limited to, videocameras 304, IR cameras and rangefinders 306, stereo (or eventrinocular) camera(s) 308 and multi-dimensional scanning lasers 310,provide multi-spectral sensory data to the main software abstractionengines 312 (after being acquired & filtered in the data acquisition andfiltering module 314). The data is used in a scene understanding module316 to carry out multiple steps such as (but not limited to) buildinghigh- and lower-resolution (laser: high-resolution; stereo-camera:lower-resolution) three-dimensional surface volumes of the scene, withsuperimposed visual and IR-spectrum color and texture video information,allowing edge-detection and volumetric object-detection algorithms toinfer what elements are in a scene, allowing the use ofshape-/color-/texture- and consistency-mapping algorithms to run on theprocessed data to feed processed information to the Kitchen CookingProcess Equipment Handling Module 318. In the module 318, software-basedengines are used for the purpose of identifying and three-dimensionallylocating the position and orientation of kitchen tools and utensils andidentifying and tagging recognizable food elements (meat, carrots,sauce, liquids, etc.) so as to generate data to let the computer buildand understand the complete scene at a particular point in time so as tobe used for next-step planning and process monitoring. Engines requiredto achieve such data and information abstraction include, but are notlimited to, grasp reasoning engines, geometry reasoning engines,physical reasoning engines and task reasoning engines. Output data fromboth engines 316 and 318 are then used to feed the scene modeler andcontent classifier 320, where the 3D world model is created with all thekey content required for executing the robotic cooking script executor.Once the fully-populated model of the world is understood, it can beused to feed the motion and handling planner 322 (if robotic-armgrasping and handling are necessary, the same data can be used todifferentiate and plan for grasping and manipulating food and kitchenitems depending on the required grip and placement) to allow forplanning motions and trajectories for the arm(s) and attachedend-effector(s) (grippers, multi-fingered hands). A follow-on ExecutionSequence planner 324 creates the proper sequencing of task-basedcommands for all individual robotic/automated kitchen elements, whichare then used by the robotic kitchen actuation systems 326. The entiresequence above is repeated in a continuous closed loop during therobotic recipe-script execution and monitoring phase.

FIG. 7A depicts the standardized kitchen 50 which in this case plays therole of the chef-studio, in which the human chef 49 carries out therecipe creation and execution while being monitored by the multi-modalsensor systems 66, so as to allow the creation of a recipe-script.Within the standardized kitchen, are contained multiple elementsnecessary for the execution of a recipe, including the main cookingmodule 350, which includes such as equipment as utensils 360, a cooktop362, a kitchen sink 358, a dishwasher 356, a table-top mixer and blender(also referred to as a “kitchen blender”) 352, an oven 354 and arefrigerator/freezer combination unit 353.

FIG. 7B depicts the standardized kitchen 50 which in this case isconfigured as the standardized robotic kitchen, in which a dual-armrobotics system with vertical telescoping and rotating torso joint 360,outfitted with two arms 70 and two wristed and fingered hands 72,carries out the recipe replication processes defined in therecipe-script. The multi-modal sensor systems 66 continually monitor therobotically executed cooking steps in the multiple stages of the recipereplication process.

FIG. 7C depicts the systems involved in the creation of a recipe-scriptby monitoring a human chef 49 during the entire recipe executionprocess. The same standardized kitchen 50 is used in a chef studio mode,with the chef able to operate the kitchen from either side of thework-module. Multi-modal sensors 66 monitor and collect data, as well asthrough the haptic gloves 370 worn by the chef and instrumented cookware372 and equipment, relaying all collected raw data wirelessly to aprocessing computer 16 for processing and storage.

FIG. 7D depicts the systems involved in a standardized kitchen 50 forthe replication of a recipe script 19 through the use of a dual-armsystem with telescoping and rotating torso 374, comprised of two arms72, two robotic wrists 71 and two multi-fingered hands 72 with embeddedsensory skin and point-sensors. The robotic dual-arm system uses theinstrumented arms and hands with a cooking utensil and an instrumentedappliance and cookware (pan in this image) on a cooktop 12, whileexecuting a particular step in the recipe replication process, whilebeing continuously monitored by the multi-modal sensor units 66 toensure the replication process is carried out as faithfully as possibleto that created by the human chef. All data from the multi-modal sensors66, dual-arm robotics system comprised of torso 74, arms 72, wrists 71and multi-fingered hands 72, utensils, cookware and appliances, iswirelessly transmitted to a computer 16, where it is processed by anonboard processing unit 16 in order to compare and track the replicationprocess of the recipe to as faithfully as possible follow the criteriaand steps as defined in the previously created recipe script 19 andstored in media 18.

FIG. 7E is a block diagram depicting the stepwise flow and methods 376to ensure that there are control or verification points during therecipe replication process based on the recipe-script when executed bythe standardized robotic kitchen 50, that ensures as nearly identical aspossible a cooking result for a particular dish as executed by thestandardized robotic kitchen 50, when compared to the dish prepared bythe human chef 49. Using a recipe 378, as described by the recipe-scriptand executed in sequential steps in the cooking process 380, thefidelity of execution of the recipe by the robotic kitchen 50 willdepend largely on considering the following main control items. Keycontrol items include the process of selecting and utilizing astandardized portion amount and shape of a high-quality andpre-processed ingredient 381, the use of standardized tools andutensils, cook-ware with standardized handles to ensure proper andsecure grasping with a known orientation 383, standardized equipment 385(oven, blender, fridge, fridge, etc.) in the standardized kitchen thatis as identical as possible when comparing the chef studio kitchen wherethe human chef 49 prepares the dish and the standardized robotic kitchen50, location and placement 384 for ingredients to be used in the recipe,and ultimately a pair of robotic arms, wrists and multi-fingered handsin a kitchen module 382 continually monitored by sensors withcomputer-controlled actions to ensure successful execution of each stepin every stage of the replication process of the recipe-script for aparticular dish. In the end the task of ensuring an identical result 386is the ultimate goal for the standardized robotic kitchen 50.

FIG. 8A is a block diagram illustrating one embodiment of a recipeconversion algorithm module 400 between the chef's movements and therobotic replication movements. A recipe algorithm conversion module 404converts the captured data from the chef's movements in the chef studio44 into a machine-readable and machine-executable language 406 forinstructing the robotic arms 70 and the robotic hands 72 to replicate afood dish prepared by the chef's movement in the robotic kitchen 48. Inthe chef studio 44, the computer 16 captures and records the chef'smovements based on the sensors on a glove 26 that the chef wears,represented by a plurality of sensors S₀, S₁, S₂, S₃, S₄, S₅, S₆ . . .S_(n) in the vertical columns, and the time increments t₀, t₁, t₂, t₃,t₄, t₅, t₆ . . . t_(end) in the horizontal rows, in a table 408. At timet₀, the computer 16 records the xyz coordinate positions from the sensordata received from the plurality of sensors S₀, S₁, S₂, S₃, S₄, S₅, S₆ .. . S_(n). At time t₁, the computer 16 records the xyz coordinatepositions from the sensor data received from the plurality of sensorsS₀, S₁, S₂, S₃, S₄, S₅, S₆ . . . S_(n). At time t₂, the computer 16records the xyz coordinate positions from the sensor data received fromthe plurality of sensors S₀, S₁, S₂, S₃, S₄, S₅, S₆ . . . S_(n). Thisprocess continues until the entire food preparation is completed at timet_(end). The duration for each time units t₀, t₁, t₂, t₃, t₄, t₅, t₆ . .. t_(end) is the same. As a result of the captured and recorded sensordata, the table 408 shows any movements from the sensors S₀, S₁, S₂, S₃,S₄, S₅, S₆ . . . S_(n) in the glove 26 in xyz coordinates, which wouldindicate the differentials between the xyz coordinate positions for onespecific time relative to the xyz coordinate positions for the nextspecific time. Effectively, the table 408 records how the chef'smovements change over the entire food preparation process from the starttime, t₀, to the end time, t_(end). The illustration in this embodimentcan be extended to two gloves 26 with sensors which the chef 49 wears tocapture the movements while preparing a food dish. In the robotickitchen 48, the robotic arms 70 and the robotic hands 72 replicate therecorded recipe from the chef studio 44, which is then converted torobotic instructions, where the robotic arms 70 and the robotic hands 72replicate the food preparation of the chef 49 according to the timeline416. The robotic arms 70 and hands 72 carry out the food preparationwith the same xyz coordinate positions, at the same speed, with the sametime increments from the start time, t₀, to the end time, t_(end), asshown in the timeline 416.

In some embodiments a chef performs the same food preparation operationmultiple times, yielding values of the sensor reading, and parameters inthe corresponding robotic instructions that vary somewhat from one timeto the next. The set of sensor readings for each sensor across multiplerepetitions of the preparation of the same food dish provides adistribution with a mean, standard deviation and minimum and maximumvalues. The corresponding variations on the robotic instructions (alsocalled the effector parameters) across multiple executions of the samefood dish by the chef also define distributions with mean, standarddeviation, minimum and maximum values. These distributions may be usedto determine the fidelity (or accuracy) of subsequent robotic foodpreparations.

In one embodiment the estimated average accuracy of a robotic foodpreparation operation is given by:

${A( {C,R} )} = {1 - {\frac{1}{n}{\sum\limits_{{n = 1},{\ldots\mspace{14mu} n}}\frac{{c_{i} - p_{i}}}{\max( {{c_{i,t} - p_{i,t}}} }}}}$

Where C represents the set of Chef parameters (1^(st) through n^(th))and R represents the set of Robotic Apparatus parameters(correspondingly (1^(st) through n^(th)). The numerator in the sumrepresents the difference between robotic and chef parameters (i.e. theerror) and the denominator normalizes for the maximal difference). Thesum gives the total normalized cumulative error

$( {i.e.{\sum\limits_{{n = 1},{\ldots\mspace{14mu} n}}\frac{{c_{i} - p_{i}}}{\max( {{c_{i,t} - p_{i,t}}} }}} )$and multiplying by 1/n gives the average error. The complement of theaverage error corresponds to the average accuracy.

Another version of the accuracy calculation weighs the parameters forimportance, where each coefficient (each α_(i)) represents theimportance of the i^(th) parameter, the normalized cumulative error is

$\sum\limits_{{n = 1},{\ldots\mspace{14mu} n}}\frac{a_{i}{{c_{i} - p_{i}}}}{\max( {{c_{i,t} - p_{i,t}}} }$and the estimated average accuracy is given by:

${A( {C,R} )} = {1 - {( {\sum\limits_{{n = 1},{\ldots\mspace{14mu} n}}\frac{a_{i}{{c_{i} - p_{i}}}}{\max( {{c_{i,t} - p_{i,t}}} }} )/{\sum\limits_{{i = 1},{\ldots\mspace{14mu} n}}\alpha_{i}}}}$

FIG. 8B is a block diagram illustrating the pair of gloves 26 a and 26 bwith sensors worn by the chef 49 for capturing and transmitting thechef's movements. In this illustrative example, which is intended toshow one example without limiting effects, a right hand glove 26 aIncludes 25 sensors to capture the various sensor data points D1, D2,D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13, D14, D15, D16, D17, D18,D19, D20, D21, D22, D23, D24, and D25, on the glove 26 a, which may haveoptional electronic and mechanical circuits 420. A left hand glove 26 bIncludes 25 sensors to capture the various sensor data points D26, D27,D28, D29, D30, D31, D32, D33, D34, D35, D36, D37, D38, D39, D40, D41,D42, D43, D44, D45, D46, D47, D48, D49, D50, on the glove 26 b, whichmay have optional electronic and mechanical circuits 422.

FIG. 8C is a block diagram illustrating robotic cooking execution stepsbased on the captured sensory data from the chef's gloves 26 a and 26 b.In the chef studio 44, the chef 49 wears gloves 26 a and 26 b withsensors for capturing the food preparation process, where the sensordata are recorded in a table 430. In this example, the chef 49 iscutting a carrot with a knife in which each slice of the carrot is about1 centimeter in thickness. These action primitives by the chef 49, asrecorded by the gloves 26 a, 26 b, may constitute a mini-manipulation432 that take place over time slots 1, 2, 3 and 4. The recipe algorithmconversion module 404 is configured to convert the recorded recipe filefrom the chef studio 44 to robotic instructions for operating therobotic arms 70 and the robotic hands 72 in the robotic kitchen 28according to a software table 434. The robotic arms 70 and the robotichands 72 prepare the food dish with control signals 436 for themini-manipulation, as pre-defined in the mini-manipulation library 116,of cutting the carrot with knife in which each slice of the carrot isabout 1 centimeter in thickness. The robotic arms 70 and the robotichands 72 operate with the same xyz coordinates 438 and with possiblereal-time adjustment on the size and shape of a particular carrot bycreating a temporary three-dimensional model 440 of the carrot from thereal-time adjustment devices 112

In order to operate a mechanical robotic mechanism such as the onesdescribed in the embodiments of this invention, a skilled artisanrealizes that many mechanical and control problems need to be addressed,and the literature in robotics describes methods to do just that. Theestablishment of static and/or dynamic stability in a robotics system isan important consideration. Especially for robotic manipulation, dynamicstability is a strongly desired property, in order to prevent accidentalbreakage or movements beyond those desired or programmed. Dynamicstability is illustrated in FIG. 8D relative to equilibrium. Here the“equilibrium value” is the desired state of the arm (i.e. the arm movesto exactly where it was programmed to move to, with deviations caused byany number of factors such as inertia, centripetal or centrifugalforces, harmonic oscillations, etc. A dynamically-stable system is onewhere variations are small and dampen out over time, as represented by acurved line 450. A dynamically unstable system is one where variationsfail to dampen and can increase over time, as depicted by a curved line452. And the worst situation is when the arm is statically unstable(e.g. it cannot hold the weight of whatever it is grasping), and falls,or it fails to recover from any deviation from the programmed positionand/or path, as illustrated by a curved line 454. For additionalinformation on planning (forming sequences of mini-manipulations, orrecovering when something goes wrong), Garagnani, M. (1999) “Improvingthe Efficiency of Processed Domain-axioms Planning”, Proceedings ofPLANSIG-99, Manchester, England, pp. 190-192, which this references isincorporated by reference herein in its entirety.

The cited literature addresses conditions for dynamic stability that areimported by reference into the present invention to enable properfunctioning of the robotic arms. These conditions include thefundamental principle for calculating torque to the joints of a roboticarm:

${\overset{->}{T} = {{{M( \overset{->}{q} )}\frac{d^{2}\overset{->}{q}}{{dt}^{2}}} + {{C( {\overset{->}{q},\frac{d\overset{->}{q}}{dt}} )}d\overset{->}{q}}}},{+ {G( \overset{->}{q} )}}$

where T is the torque vector (T has n components, each corresponding toa degree of freedom of the robotic arm), M is the inertial matrix of thesystem (M is a positive semi-definite n-by-n matrix), C is a combinationof centripetal and centrifugal forces, also an n-by-n matrix, G(q) isthe gravity vector, and q is the position vector. And they includefinding stable points and minima, e.g. via the LaGrange equation if therobotic positions (x's) can be described by twice-differentiablefunctions (y's).

In order for the system comprised of the robotic arms and hands/grippersto be stable, it is important that the system be properly designed andbuilt and have an appropriate sensing and control system which operateswithin the boundary of acceptable performance. The reason that this isimportant is that one wants to achieve the best (highest speed withhighest position/velocity and force/torque tracking and all under stableconditions) performance possible given the physical system and what itscontroller is asking it to do.

When one speaks of proper design, the notion is one of achieving properobservability and controllability of the system. Observability impliesthat the key variables of the system (joint/finger positions andvelocities, forces and torques) are measurable by the system, whichimplies one needs to have the ability to sense these variables, which inturn implies the presence and use of the proper sensing devices(internal or external). Controllability implies that one (computer inthis case) have the ability to shape or control the key axes of thesystem based on observed parameters from internal/external sensors; thisusually implies an actuator or direct/indirect control over a certainparameter by way of a motor or other computer-controlled actuationsystem. The ability to make the system as linear in its response aspossible, thereby negating the detrimental effects of nonlinearities(stiction, backlash, hysteresis, etc.), allows for control schemes likePID gain-scheduling and nonlinear controllers like sliding-mode controlto guarantee system stability and performance even in the light ofsystem-modeling uncertainties (errors in mass/inertia estimates,dimensional geometry discretization, sensor/torque discretizationanomalies, etc.) which are always present in any higher-performancecontrol system.

Furthermore, the use of a proper computing and sampling system issignificant, as the system's ability to follow rapid motions with acertain maximum frequency content is clearly related to what controlbandwidth (closed-loop sampling rate of the computer control system) theentire system is able to achieve and thus the frequency-response(ability to track motions of certain speeds and motion-frequencycontent) the system is able to exhibit.

All the above characteristics are significant when it comes to ensuringthat a highly redundant system can actually carry out the complex anddexterous tasks a human chef requires for a successful recipe-scriptexecution, in both a dynamic and a stable fashion.

Machine learning in the context of robotic manipulation of relevance tothe invention can involve well known methods for parameter adjustment,such as reinforcement learning. An alternate and preferred embodimentfor this invention is a different and more appropriate learningtechnique for repetitive complex actions such as preparing and cooking ameal with multiple steps over time, namely case-based learning.Case-based reasoning, also known as analogical reasoning, has beendeveloped over time.

As a general overview, case-based reasoning comprises the followingsteps:

A. Constructing and remembering cases. A case is a sequence of actionswith parameters that are successfully carried out to achieve anobjective. The parameters include distances, forces, directions,positions, and other physical or electronic measures whose values arerequired to successfully carry out the task (e.g. a cooking operation).First,

-   -   1. storing aspects of the problem that was just solved together        with:    -   2. the method(s) and optionally intermediate steps to solve the        problem and its parameter values, and    -   3. (typically) storing the final outcome.        B. Applying cases (at a later point of time)    -   4. Retrieving one or more stored cases whose problems bear        strong similarity to the new problem,    -   5. Optionally adjusting the parameters from the retrieved        case(s) to apply to the current case (e.g. an item may weigh        somewhat more, and hence a somewhat stronger force is needed to        lift it),    -   6. Using the same methods and steps from the case(s) with the        adjusted parameters (if needed) at least in part to solve the        new problem.        Hence, case-based reasoning consists of remembering solutions to        past problems and applying them with possible parametric        modification to new very similar problems. However, in order to        apply case-based reasoning to the robotic manipulation        challenge, something more is needed. Variation in one parameter        of the solution plan will cause variation in one or more coupled        parameters. This requires transformation of the problem        solution, not just application. We call the new process        case-based robotic learning since it generalizes the solution to        a family of close solutions (those corresponding to small        variations in the input parameters—such as exact weight, shape        and location of the input ingredients). Case-based robotic        learning operates as follows:        C. Constructing, remembering and transforming robotic        manipulation cases    -   1. Storing aspects of the problem that was just solved together        with:    -   2. The value of the parameters (e.g. the inertial matrix,        forces, etc. from equation 1),    -   3. Perform perturbation analysis by varying the parameter(s)        pertinent to the domain (e.g. in cooking, vary the weight of the        materials or their exact starting position), to see how much        parameter values can vary and still obtain the desired results,    -   4. Via perturbation analysis on the model, record which other        parameter values will change (e.g. forces) and by how much they        should change, and    -   5. If the changes are within operating specification of the        robotic apparatus, store the transformed solution plan (with the        dependencies among parameters and projected change calculations        for their values).        D. Applying cases (at a later point of time)    -   6. Retrieve one or more stored cases with the transformed exact        values (now ranges, or calculations for new values depending on        values of the input parameters), but still whose initial        problems bear strong similarity to the new problem, including        parameter values and value ranges, and    -   7. Use the transformed methods and steps from the case(s) at        least in part to solve the new problem.        As the chef teaches the robot (the two arms and the sensing        devices, such as haptic feedback from fingers, force-feedback        from joints, and one or more observation cameras), the robot        learns not only the specific sequence of movements, and time        correlations, but also the family of small variations around the        chef's movements to be able to prepare the same dish regardless        of minor variations in the observable input parameters—and thus        it learns a generalized transformed plan, giving it far greater        utility than rote memorization. For additional information on        case-based reasoning and learning, see materials by Leake, 1996        Book, Case-Based Reasoning: Experiences, Lessons and Future        Directions,        http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=4068324&fileld=S0269888        900006585dl.acm.org/citation.cfm?id=524680; Carbonell, 1983,        Learning by Analogy: Formulating and Generalizing Plans from        Past Experience,        http://link.springer.com/chapter/10.1007/978-3-662-12405-5_5,        which these references are incorporated by reference herein in        their entireties.

As depicted in FIG. 8E, the process of cooking requires a sequence ofsteps that are referred to as a plurality of stages S₁, S₂, S₃ . . .S_(j) . . . S_(n) of food preparation, as shown in a timeline 456. Thesemay require strict linear/sequential ordering or some may be performedin parallel; either way we have a set of stages {S₁, S₂, . . . , S_(i),. . . , S_(n)}, all of which must be completed successfully to achieveoverall success. If the probability of success for each stage isP(s_(i)) and there are n stages, then the probability of overall successis estimated by the product of the probability of success at each stage:

${P(S)} = {\prod\limits_{S_{i} \in S}{P( s_{i} )}}$

A person of skill in the art will appreciate that the probability ofoverall success can be low even if the probability of success ofindividual stages is relatively high. For instance, given 10 stages anda probability of success of each stage being 90%, the probability ofoverall success is (0.9)¹⁰=0.28 or 28%.

A stage in preparing a food dish comprises one or moremini-manipulations, where each mini-manipulation comprises one or morerobotic actions leading to a well-defined intermediate result. Forinstance, slicing a vegetable can be a mini-manipulation consisting ofgrasping the vegetable with one hand, grasping a knife with the other,and applying repeated knife movements until the vegetable is sliced. Astage in preparing a dish can comprise one or multiple slicingmini-manipulations.

The probability of success formula applies equally well at the level ofstages and at the level of mini-manipulations, so long as eachmini-manipulation is relatively independent of other mini-manipulations.

In one embodiment, in order to mitigate the problem of reduced certaintyof success due to potential compounding errors, standardized methods formost or all of the mini-manipulations in all of the stages arerecommended. Standardized operations are ones that can bepre-programmed, pre-tested, and if necessary pre-adjusted to select thesequence of operations with the highest probability of success. Hence,if the probability of standardized methods via the mini-manipulationswithin stages is very high, so will be the overall probability ofsuccess of preparing the food dish, due to the prior work, until all ofthe steps have been perfected and tested. For instance, to return to theabove example, if each stage utilizes reliable standardized methods, andits success probability is 99% (instead of 90% as in the earlierexample), then the overall probability of success will be(0.99)¹⁰=90.4%, assuming there are 10 stages as before. This is clearlybetter than 28% probability of an overall correct outcome.

In another embodiment, more than one alternative method is provided foreach stage, wherein, if one alternative fails, another alternative istried. This requires dynamic monitoring to determine the success orfailure of each stage, and the ability to have an alternate plan. Theprobability of success for that stage is the complement of theprobability of failure for all of the alternatives, which mathematicallyis written as:

${P( s_{i} \middle| {A( s_{i} )} )} = {1 - {\prod\limits_{a_{j} \in {A{(s_{i})}}}( {1 - {P( s_{i} \middle| a_{j} )}} )}}$

In the above expression s_(i) is the stage and A(s_(i)) is the set ofalternatives for accomplishing S_(i). The probability of failure for agiven alternative is the complement of the probability of success forthat alternative, namely 1−P(s_(i)|a_(j)), and the probability of allthe alternatives failing is the product in the above formula. Hence, theprobability that not all will fail is the complement of the product.Using the method of alternatives, the overall probability of success canbe estimated as the product of each stage with alternatives, namely:

${P(S)} = {\prod\limits_{S_{i} \in S}{P( s_{i} \middle| {A( s_{i} )} )}}$

With this method of alternatives, if each of the 10 stages had 4alternatives, and the expected success of each alternative for eachstage was 90%, then the overall probability of success would be(1−(1−(0.9))⁴)¹⁰=0.99 or 99% versus just 28% without the alternatives.The method of alternatives transforms the original problem from a chainof stages with multiple single points of failure (if any stage fails) toone without single points of failure, since all the alternatives wouldneed to fail in order for any given stage to fail, providing more robustoutcomes.

In another embodiment, both standardized stages comprising standardizedmini-manipulations, and alternate means of the food dish preparationstages are combined, yielding even more robust behavior. In such a case,the corresponding probability of success can be very high, even ifalternatives are only present for some of the stages ormini-manipulations.

In another embodiment only the stages with lower probability of successare provided alternatives, in case of failure, for instance stages forwhich there is no very reliable standardized method, or for which thereis potential variability, e.g. depending on odd-shaped materials. Thisembodiment reduces the burden of providing alternatives to all stages.

FIG. 8F is a graphical diagram showing the probability of overallsuccess (y-axis) as a function of the number of stages needed to cook afood dish (x-axis) for a first curve 458 illustrating a non-standardizedkitchen 458 and a second curve 459 illustrating the standardized kitchen50. In this example, the assumption made is that the individualprobability of success per food preparation stage was 90% for anon-standardized operation and 99% for a standardized pre-programmedstage. The compounded error is much worse in the former case, as shownin the curve 458 compared to the curve 459.

FIG. 8G is a block diagram illustrating the execution of a recipe 460with multi-stage robotic food preparation with mini-manipulations andaction primitives. Each food recipe 460 can be divided into a pluralityof food preparation stages: a first food preparation stage S₁ 470, asecond food preparation stage S₂ . . . an n-stage food preparation stageS_(n) 490, as executed by the robotic arms 70 and the robotic hands 72.The first food preparation stage S₁ 470 comprises one or moremini-manipulations MM₁ 471, MM₂ 472, and MM₃ 473. Each mini-manipulationincludes one or more action primitives which obtains a functionalresult. For example, the first mini-manipulation MM₁ 471 includes afirst action primitive AP₁ 474, a second action primitive AP₂ 475, and athird action primitive AP₃ 475, which then achieves a functional result477. The one or more mini-manipulations MM₁ 471, MM₂ 472, MM₃ 473 in thefirst stage S₁ 470 then accomplish a stage result 479. The combinationof one or more food preparation stage S₁ 470, the second foodpreparation stage S₂ and the n-stage food preparation stage S_(n) 490produces substantially the same or the same result by replicating thefood preparation process of the chef 49 as recorded in the chef studio44.

A predefined mini-manipulation is available to achieve each functionalresult (e.g., the egg is cracked). Each mini-manipulation comprises of acollection of action primitives which act together to accomplish thefunctional result. For example, the robot may begin by moving its handtowards the egg, touching the egg to localize its position and verifyits size, and executing the movements and sensing actions necessary tograsp and lift the egg into the known and predetermined configuration.

Multiple mini-manipulations may be collected into stages such as makinga sauce for convenience in understanding and organizing the recipe. Theend result of executing all of the mini-manipulations to complete all ofthe stages is that a food dish has been replicated with a consistentresult each time.

FIG. 9A is a block diagram illustrating an example of the robotic hand72 with five fingers and a wrist with RGB-D sensor, camera sensors andsonar sensor capabilities for detecting and moving a kitchen tool, anobject, or an item of kitchen equipment. The palm of the robotic hand 72includes an RGB-D sensor 500, a camera sensor or a sonar sensor 504 f.Alternatively, the palm of the robotic hand 450 includes both the camerasensor and the sonar sensor. The RGB-D sensor 500 or the sonar sensor504 f is capable of detecting the location, dimensions and shape of theobject to create a three-dimensional model of the object. For example,the RGB-D sensor 500 uses structured light to capture the shape of theobject, three-dimensional mapping and localization, path planning,navigation, object recognition and people tracking. The sonar sensor 504f uses acoustic waves to capture the shape of the object. In conjunctionwith the camera sensor 452 and/or the sonar sensor 454, the video camera66 placed somewhere in the robotic kitchen, such as on a railing, or ona robot, provides a way to capture, follow, or direct the movement ofthe kitchen tool as used by the chef 49, as illustrated in FIG. 7A. Thevideo camera 66 is positioned at an angle and some distance away fromthe robotic hand 72, and therefore provides a higher-level view of therobotic hand's 72 gripping of the object, and whether the robotic handhas gripped or relinquished/released the object. A suitable example ofRGB-D (a red light beam, a green light beam, a blue light beam, anddepth) sensor is the Kinect system by Microsoft, which features an RGBcamera, depth sensor and multi-array microphone running on software,which provide full-body 3D motion capture, facial recognition and voicerecognition capabilities.

The robotic hand 72 has the RGB-D sensor 500 placed in or near themiddle of the palm for detecting the distance and shape of an object, aswell as the distance of the object, and for handling a kitchen tool. TheRGB-D sensor 500 provides guidance to the robotic hand 72 in moving therobotic hand 72 toward the direction of the object and to make necessaryadjustments to grab an object. Second, a sonar sensor 502 f and/or atactile pressure sensor are placed near the palm of the robotic hand 72,for detecting the distance and shape, and subsequent contact, of theobject. The sonar sensor 502 f can also guide the robotic hand 72 tomove toward the object. Additional types of sensors in the hand mayinclude ultrasonic sensors, lasers, radio frequency identification(RFID) sensors, and other suitable sensors. In addition, the tactilepressure sensor serves as a feedback mechanism so as to determinewhether the robotic hand 72 continues to exert additional pressure tograb the object at such point where there is sufficient pressure tosafely lift the object. In addition, the sonar sensor 502 f in the palmof the robotic hand 72 provides a tactile sensing function to grab andhandle a kitchen tool. For example, when the robotic hand 72 grabs aknife to cut beef, the amount of pressure that the robotic hand exertson the knife and applies to the beef can be detected by the tactilesensor when the knife finishes slicing the beef, i.e. when the knife hasno resistance, or when holding an object. The pressure distributed isnot only to secure the object, but also not to break it (e.g. an egg).

Furthermore, each finger on the robotic hand 72 has haptic vibrationsensors 502 a-e and sonar sensors 504 a-e on the respective fingertips,as shown by a first haptic vibration sensor 502 a and a first sonarsensor 504 a on the fingertip of the thumb, a second haptic vibrationsensor 502 b and a second sonar sensor 504 b on the fingertip of theindex finger, a third haptic vibration sensor 502 c and a third sonarsensor 504 c on the fingertip of the middle finger, a fourth hapticvibration sensor 502 d and a fourth sonar sensor 504 d on the fingertipof the ring finger, and a fifth haptic vibration sensor 502 e and afifth sonar sensor 504 e on the fingertip of the pinky. Each of thehaptic vibration sensors 502 a, 502 b, 502 c, 502 d and 502 e cansimulate different surfaces and effects by varying the shape, frequency,amplitude, duration and direction of a vibration. Each of the sonarsensors 504 a, 504 b, 504 c, 504 d and 504 e provides sensing capabilityon the distance and shape of the object, sensing capability for thetemperature or moisture, as well as feedback capability. Additionalsonar sensors 504 g and 504 h are placed on the wrist of the robotichand 72.

FIG. 9B is a block diagram illustrating one embodiment of a pan-tilthead 510 with a sensor camera 512 coupled to a pair of robotic arms andhands for operation in the standardized robotic kitchen. The pan-tilthead 510 has an RGB-D sensor 512 for monitoring, capturing or processinginformation and three-dimensional images within the standardized robotickitchen 50. The pan-tilt head 510 provides good situational awarenesswhich is independent of arm and sensor motions. The pan-tilt head 510 iscoupled to the pair of robotic arms 70 and hands 72 for executing foodpreparation processes, but the pair of robotic arms 70 and hands 72 maycause occlusions.

FIG. 9C is a block diagram illustrating sensor cameras 514 on therobotic wrists 73 for operation in the standardized robotic kitchen 50.One embodiment of the sensor cameras 514 is an RGB-D sensor thatprovides color image and depth perception mounted to the wrists 73 ofthe respective hand 72. Each of the camera sensors 514 on the respectivewrist 73 provides limited occlusions by an arm, while generally notoccluded when the robotic hand 72 grasps an object. However, the RGB-Dsensors 514 may be occluded by the respective robotic hand 72.

FIG. 9D is a block diagram illustrating an eye-in-hand 518 on therobotic hands 72 for operation in the standardized robotic kitchen 50.Each hand 72 has a sensor, such as an RGD-D sensor for providing aneye-in-hand function by the robotic hand 72 in the standardized robotickitchen 50. The eye-in-hand 518 with RGB-D sensor in each hand provideshigh image details with limited occlusions by the respective robotic arm70 and the respective robotic hand 72. However, the robotic hand 72 withthe eye-in-hand 518 may encounter occlusions when grasping an object.

FIGS. 9E-G are pictorial diagrams illustrating aspects of a deformablepalm 520 in the robotic hand 72. The fingers of a five-fingered hand arelabeled with the thumb as a first finger F1 522, the index finger as asecond finger F2 524, the middle finger as a third finger F3 526, thering finger as a fourth finger F4 528, and the little finger as a fifthfinger F5 530. The thenar eminence 532 is a convex volume of deformablematerial on the radial (the first finger F1 522) side of the hand. Thehypothenar eminence 534 is a convex volume of deformable material on theulnar (the fifth finger F5 530) side of the hand. Themetacarpophalangeal pads (MCP pads) 536 are convex deformable volumes onthe ventral (palmar) side of the metacarpophalangeal (knuckle) joints ofsecond, third, fourth and fifth fingers F2 524, F3 526, F4 528, F5 530.The robotic hand 72 with the deformable palm 520 wears a glove on theoutside with a soft human-like skin.

Together the thenar eminence 532 and hypothenar eminence 534 supportapplication of large forces from the robot arm to an object in theworking space such that application of these forces puts minimal stresson the robot hand joints (e.g., picture of the rolling pin). Extrajoints within the palm 520 themselves are available to deform the palm.The palm 520 should deform in such a way as to enable the formation ofan oblique palmar gutter for tool grasping in a way similar to a chef(typical handle grasp). The palm 520 should deform in such a way as toenable cupping, for conformable grasping of convex objects such asdishes and food materials in a manner similar to the chef, as shown by acupping posture 542 in FIG. 9G.

Joints within the palm 520 that may support these motions include thethumb carpometacarpal joint (CMC), located on the radial side of thepalm near the wrist, which may have two distinct directions of motion(flexion/extension and abduction/adduction). Additional joints requiredto support these motions may include joints on the ulnar side of thepalm near the wrist (the fourth finger F4 528 and the fifth finger F5530 CMC joints), which allow flexion at an oblique angle to supportcupping motion at the hypothenar eminence 534 and formation of thepalmar gutter.

The robotic palm 520 may include additional/different joints as neededto replicate the palm shape observed in human cooking motions, e.g., aseries of coupled flexure joints to support formation of an arch 540between the thenar and hypothenar eminences 532 and 534 to deform thepalm 520, such as when the thumb F1 522 touches the pinky finger F5 530,as illustrated in FIG. 9F.

When the palm is cupped, the thenar eminence 532, the hypothenareminence 534, and the MCP pads 536 form ridges around a palmar valleythat enable the palm to close around a small spherical object (e.g., 2cm).

The shape of the deformable palm will be described using locations offeature points relative to a fixed reference frame, as shown in FIGS. 9Hand 9I. Each feature point is represented as a vector of x, y, and zcoordinate positions over time. Feature point locations are marked onthe sensing glove worn by the chef and on the sensing glove worn by therobot. A reference frame is also marked on the glove, as illustrated inFIGS. 9H and 9I. Feature points are defined on a glove relative to theposition of the reference frame.

Feature points are measured by calibrated cameras mounted in theworkspace as the chef performs cooking tasks. Trajectories of featurepoints in time are used to match the chef motion with the robot motion,including matching the shape of the deformable palm. Trajectories offeature points from the chef's motion may also be used to inform robotdeformable palm design, including shape of the deformable palm surfaceand placement and range of motion of the joints of the robot hand.

In the embodiment as depicted in FIG. 9H, the feature points are in thehypothenar eminence 534, the thenar eminence 532, and the MCP pad 536are checkered patterns with markings that show the feature points ineach region of the palm. The reference frame in the wrist area has fourrectangles that are identifiable as a reference frame. The featurepoints (or markers) are identified in their respective locationsrelative to the reference frame. The feature points and reference framein this embodiment can be implemented underneath a glove for food safetybut transparent through the glove for detection.

FIG. 9H shows the robot hand with a visual pattern which may be used todetermine the locations of three-dimensional shape feature points 550.The locations of these shape feature points provide information aboutthe shape of the palm surface as the palm joints move and as the palmsurface deforms in response to applied forces.

The visual pattern consists of surface markings 552 on the robot hand oron a glove worn by the chef. These surface markings may be covered by afood safe transparent glove 554, but the surface markings 552 remainvisible through the glove.

When the surface markings 552 are visible in a camera image,two-dimensional feature points may be identified within that cameraimage by locating convex or concave corners within the visual pattern.Each such corner in a single camera image is a two-dimensional featurepoint.

When the same feature point is identified in multiple camera images, thethree-dimensional location of this point can be determined in acoordinate frame which is fixed with respect to the standardized robotickitchen 50. This calculation is performed based on the two-dimensionallocation of the point in each image and the known camera parameters(position, orientation, field of view, etc. . . . ).

A reference frame 556 fixed to the robotic hand 72 can be obtained usinga reference frame visual pattern. In one embodiment, the reference frame556 fixed to the robotic hand 72 comprises of an origin and threeorthogonal coordinate axes. It is identified by locating features of thereference frame's visual pattern in multiple cameras, and using knownparameters of the reference frame visual pattern and known parameters ofthe cameras to extract the origin and coordinate axes.

Three-dimensional shape feature points expressed in the coordinate frameof the food preparation station can be converted into the referenceframe of the robot hand once the reference frame of the robot hand isobserved.

The shape of the deformable palm is comprised of a vector ofthree-dimensional shape feature points, all of which are expressed inthe reference coordinate frame fixed to the hand of the robot or thechef.

As illustrated in FIG. 9I, the feature points 560 in the embodiments arerepresented by the sensors, such as Hall effect sensors, in thedifferent regions (the hypothenar eminence 534, the thenar eminence 532,and the MCP pad 536 of the palm. The feature points are identifiable intheir respective locations relative to the reference frame, which inthis implementation is a magnet. The magnet produces magnetic fieldsthat are readable by the sensors. The sensors in this embodiment areembedded underneath the glove.

FIG. 9I shows the robot hand 72 with embedded sensors and one or moremagnets 562 which may be used as an alternative mechanism to determinethe locations of three-dimensional shape feature points. One shapefeature point is associated with each embedded sensor. The locations ofthese shape feature points 560 provide information about the shape ofthe palm surface as the palm joints move and as the palm surface deformsin response to applied forces.

Shape feature point locations are determined based on sensor signals.The sensors provide an output which allows calculation of distance in areference frame which is attached to the magnet, which furthermore isattached to the hand of the robot or the chef.

The three-dimensional location of each shape feature point is calculatedbased on the sensor measurements and known parameters obtained fromsensor calibration. The shape of the deformable palm is comprised of avector of three-dimensional shape feature points, all of which areexpressed in the reference coordinate frame, which is fixed to the handof the robot or the chef. For additional information on common contactregions on the human hand and function in grasping, see the materialfrom Kamakura, Noriko, Michiko Matsuo, Harumi Ishii, Fumiko Mitsuboshi,and Yoriko Miura. “Patterns of static prehension in normal hands.”American Journal of Occupational Therapy 34, no. 7 (1980): 437-445,which this reference is incorporated by reference herein in itsentirety.

FIG. 10A is block diagram illustrating examples of chef recordingdevices 550 which the chef 49 wears in the standardized robotic kitchenenvironment 50 for recording and capturing the chef's movements duringthe food preparation process for a specific recipe. The chef recordingdevices 550 include, but are not limited to, one or more robot gloves(or robot garment) 26, a multimodal sensor unit 20 and a pair of robotglasses 552. In the chef studio system 44, the chef 49 wears the robotgloves 26 for cooking, recording, and capturing the chef's cookingmovements. Alternatively, the chef 49 may wear a robotic costume withrobotic gloves instead of just the robot gloves 26. In one embodiment,the robot glove 26, with embedded sensors, captures, records and savesthe position, pressure and other parameters of the chef's arm, hand, andfinger motions in an xyz-coordinate system with a time-stamp. The robotgloves 26 save the position and pressure of the arms and fingers of thechef 18 in a three-dimensional coordinate frame over a time durationfrom the start time to the end time in preparing a particular food dish.When the chef 49 wears the robotic gloves 26, all of the movements, theposition of the hands, the grasping motions, and the amount of pressureexerted, in preparing a food dish in the chef studio system 44, areprecisely recorded at a periodic time interval, such as every t seconds.The multimodal sensor unit(s) 20 include video cameras, IR cameras andrangefinders 306, stereo (or even trinocular) camera(s) 308 andmulti-dimensional scanning lasers 310, and provide multi-spectralsensory data to the main software abstraction engines 312 (after beingacquired and filtered in the data acquisition and filtering module 314).The multimodal sensor unit 20 generates a three-dimensional surface ortexture, and processes abstraction model-data. The data is used in ascene understanding module 316 to carry out multiple steps such as (butnot limited to) building high- and lower-resolution (laser:high-resolution; stereo-camera: lower-resolution) three-dimensionalsurface volumes of the scene, with superimposed visual and IR-spectrumcolor and texture video-information, allowing edge-detection andvolumetric object-detection algorithms to infer what elements are in ascene, allowing the use of shape-/color-/texture- andconsistency-mapping algorithms to run on the processed data to feedprocessed information to the Kitchen Cooking Process Equipment HandlingModule 318. Optionally, in addition to the robot gloves 76, the chef 49can wear a pair of robot glasses 552, which has one or more robotsensors 554 around the frame with a robot earpiece 556 and a microphone558. The robot glasses 552 provide additional vision and capturingcapabilities such as a camera for capturing video and recording imagesthat the chef 49 sees while cooking a meal. The one or more robotsensors 554 capture and record temperature and smell of the meal that isbeing prepared. The earpiece 556 and the microphone 558 capture andrecord sounds that the chef 49 hears while cooking, which may includehuman voices, sounds characteristics of frying, grilling, grinding, etc.The chef 49 may also record simultaneous voice instructions andreal-time cooking steps of the food preparation by using the earpieceand microphone 82. In this respect, the chef robot recorder devices 550record the chef's movements, speed, temperature and sound parametersduring the food preparation process for a particular food dish.

FIG. 10B is a flow diagram illustrating one embodiment of the process560 in evaluating the captured of chef's motions with robot poses,motions and forces. A database 561 stores predefined (or predetermined)grasp poses 562 and predefined hand motions by the robotic arms 72 andthe robotic hands 72, which are weighted by importance 564, labeled withpoints of contact 565, and stored contact forces 565. At operation 567,the chef movements recording module 98 is configured to capture thechef's motions in preparing a food dish based in part on the predefinedgrasp poses 562 and the predefined hand motions 563. At operation 568,the robotic food preparation engine 56 is configured to evaluate therobot apparatus configuration for its ability to achieve poses, motionsand forces, and to accomplish mini-manipulations. Subsequently, therobot apparatus configuration undergoes an iterative process 569 inassessing the robot design parameters 570, adjusting design parametersto improve the score and performance 571, and modifying the robotapparatus configuration 572.

FIG. 11 is block diagram illustrating one embodiment of a side view ofthe robotic arm 70 for use with the standardized robotic kitchen system50 in the household robotic kitchen 48. In other embodiments, one ormore of the robotic arms 70, such as one arm, two arms, three arms, fourarms, or more, can be designed for operation in the standardized robotickitchen 50. The one or more software recipe files 46 from the chefstudio system 44, which store a chef's arm, hand, and finger movementsduring food preparation, can be uploaded and converted into roboticinstructions to control the one or more robotic arms 70 and the one ormore robotic hands 72 to emulate the chef's movements for preparing afood dish that the chef has prepared. The robotic instructions controlthe robotic apparatus to replicate the precise movements of the chef inpreparing the same food dish. Each of the robotic arms 70 and each ofthe robotic hands 72 may also include additional features and tools,such as a knife, a fork, a spoon, a spatula, other types of utensils, orfood preparation instruments to accomplish the food preparation process.

FIGS. 12A-C are block diagrams illustrating one embodiment of a kitchenhandle 580 for use with the robotic hand 72 with the palm 520. Thedesign of the kitchen handle 580 is intended to be universal (orstandardized) so that the same kitchen handle 580 can attach to any typeof kitchen utensils or tools, e.g. a knife, a spatula, a skimmer, aladle, a draining spoon, a turner, etc. Different perspective views ofthe kitchen handle 580 are shown in FIGS. 12A-B. The robotic hand 72grips the kitchen handle 580 as shown in FIG. 12C. Other types ofstandardized (or universal) kitchen handles may be designed withoutdeparting from the spirit of the present invention.

FIG. 13 is a pictorial diagram illustrating an example robotic hand 600with tactile sensors 602 and distributed pressure sensors 604. Duringthe food preparation process, the robotic apparatus uses touch signalsgenerated by sensors in the fingertips and the palms of a robot's handsto detect force, temperature, humidity and toxicity as the robotreplicates step-by-step movements and compares the sensed values withthe tactile profile of the chef's studio cooking program. Visual sensorshelp the robot to identify the surroundings and take appropriate cookingactions. The robotic apparatus analyzes the image of the immediateenvironment from the visual sensors and compares it with the saved imageof the chef's studio cooking program, so that appropriate movements aremade to achieve identical results. The robotic apparatus also usesdifferent microphones to compare the chef's instructional speech tobackground noise from the food preparation processes to improverecognition performance during cooking. Optionally, the robot may havean electronic nose (not shown) to detect odor or flavor and surroundingtemperature. For example, the robotic hand 600 is capable ofdifferentiating a real egg by surface texture, temperature and weightsignals generated by haptic sensors in the fingers and palm, and is thusable to apply the proper amount of force to hold an egg without breakingit, as well as performing a quality check by shaking and listening forsloshing, cracking the egg and observing and smelling the yolk andalbumen to determine the freshness. The robotic hand 600 then may takeaction to dispose of a bad egg or select a fresh egg. The sensors 602and 604 on hands, arms, and head enable the robot to move, touch, seeand hear to execute the food preparation process using external feedbackand obtain a result in the food dish preparation that is identical tothe chef's studio cooking result.

FIG. 14 is a pictorial diagram illustrating an example of a sensingcostume 620 (for the chef 49 to wear at the standardized robotic kitchen50. During the food preparation of a food dish, as recorded by asoftware file 46, the chef 49 wears the sensing costume 620 forcapturing the real-time chef's food preparation movements in a timesequence. The sensing costume 620 may include, but is not limited to, ahaptic suit 622 (shown one full-length arm and hand costume)[again, nonumber like that in there], haptic gloves 624, a multimodal sensor(s)626 [no such number], a head costume 628. The haptic suit 622 withsensors is capable of capturing data from the chef's movements andtransmitting captured data to the computer 16 to record the xyzcoordinate positions and pressure of human arms 70 and hands/fingers 72in the XYZ-coordinate system with a time-stamp. The sensing costume 620also senses and the computer 16 records the position, velocity andforces/torques and endpoint contact behavior of human arms 70 andhands/fingers 72 in a robot-coordinate frame with and associates themwith a system timestamp, for correlating with the relative positions inthe standardized robotic kitchen 50 with geometric sensors (laser, 3Dstereo, or video sensors). The haptic glove 624 with sensors is used tocapture, record and save force, temperature, humidity, and toxicitysignals detected by tactile sensors in the gloves 624. The head costume628 includes feedback devices with vision camera, sonar, laser, or radiofrequency identification (RFID) and a custom pair of glasses that areused to sense, capture, and transmit the captured data to the computer16 for recording and storing images that the chef 48 observes during thefood preparation process. In addition, the head costume 628 alsoincludes sensors for detecting the surrounding temperature and smellsignatures in the standardized robotic kitchen 50. Furthermore, the headcostume 628 also includes an audio sensor for capturing the audio thatthe chef 49 hears, such as sound characteristics of frying, grinding,chopping, etc.

FIGS. 15A-B are pictorial diagrams illustrating one embodiment of athree-finger haptic glove 630 with sensors for food preparation by thechef 49 and an example of a three-fingered robotic hand 640 withsensors. The embodiment illustrated herein shows the simplified robotichand 640 which has less than five fingers for food preparation.Correspondingly, the complexity in the design of the simplified robotichand 640 would be significantly reduced, as well as the cost tomanufacture the simplified robotic hand 640. Two finger grippers orfour-finger robotic hands, with or without an opposing thumb, are alsopossible alternate implementations. In this embodiment, the chef's handmovements are limited by the functionalities of the three fingers,thumb, index finder and middle finger, where each finger has a sensor632 for sensing data of the chef's movement with respect to force,temperature, humidity, toxicity or tactile-sensation. The three-fingerhaptic glove 630 also includes point sensors or distributed pressuresensors in the palm area of the three-finger haptic glove 630. Thechef's movements in preparing a food dish wearing the three-fingerhaptic glove 630 using the thumb, the index finger, and the middlefingers are recorded in a software file. Subsequently, thethree-fingered robotic hand 640 replicates the chef's movements from theconverted software recipe file into robotic instructions for controllingthe thumb, the index finger and the middle finger of the robotic hand640 while monitoring sensors 642 b on the fingers and sensors 644 on thepalm of the robotic hand 640. The sensors 642 include a force,temperature, humidity, toxicity or tactile sensor, while the sensors 644can be implemented with point sensors or distributed pressure sensors.

FIG. 16 is a block diagram illustrating a creation module 650 of amini-manipulation library database and an execution module 660 of themini-manipulation library database. The creation module 60 of themini-manipulation database library is a process of creating, testingvarious possible combinations, and selecting an optimalmini-manipulation to achieve a specific functional result. One objectiveof the creation modules 60 is to explore all different possiblecombinations in performing a specific mini-manipulation and predefine alibrary of optimal mini-manipulations for subsequent execution by therobotic arms 70 and the robotic hands 72 in preparing a food dish. Thecreation module 650 of the mini-manipulation library can also be used asa teaching method for the robotic arms 70 and the robotic hands 72 tolearn about the different food preparation functions from themini-manipulation library database. The execution modules 660 of themini-manipulations library database is configured to provide a range ofmini-manipulation functions which the robotic apparatus can access andexecute from the mini-manipulations library database containing a firstmini-manipulation MM₁ with a first functional outcome 662, a secondmini-manipulation MM₂ with a second functional outcome 664, a thirdmini-manipulation MM₃ with a third functional outcome 666, a fourthmini-manipulation MM₄ with a fourth functional outcome 668, and a fifthmini-manipulation MM₅ with a fifth functional outcome 670, during theprocess of preparing a food dish.

FIG. 17A is a block diagram illustrating a sensing glove 680 used by thechef 49 to sense and capture the chef's movements while preparing a fooddish. The sensing glove 680 has a plurality of sensors 682 a, 682 b, 682c, 682 d, 682 e on each of the fingers, and a plurality of sensors 682f, 682 g, in the palm area of the sensing glove 680. In one embodiment,the at least 5 pressure sensors 682 a, 682 b, 682 c, 682 d, 682 e insidethe soft glove are used for capturing and analyzing the chef's movementsduring all hand manipulations. The plurality of sensors 682 a, 682 b,682 c, 682 d, 682 e, 682 f, and 682 g in this embodiment are embedded inthe sensing glove 680 but transparent to the material of the sensingglove 680 for external sensing. The sensing glove 680 may have featurepoints associated with the plurality of sensors 682 a, 682 b, 682 c, 682d, 682 e, 682 f, 682 g that reflect the hand curvature (or relief) ofvarious higher and lower points in the sensing glove 680. The sensingglove 680, which is placed over the robotic hand 72, is made of softmaterials that emulate the compliance and shape of human skin.Additional description elaborating on the robotic hand 72 can be foundin FIG. 9A.

The robotic hand 72 includes a camera sensor 684, such as an RGB-Dsensor, an imaging sensor or a visual sensing device, placed in or nearthe middle of the palm for detecting the distance and shape of anobject, as well as the distance of the object, and for handling akitchen tool. The imaging sensor 682 f provides guidance to the robotichand 72 in moving the robotic hand 72 towards the direction of theobject and to make necessary adjustments to grab an object. In addition,a sonar sensor, such as a tactile pressure sensor, may be placed nearthe palm of the robotic hand 72, for detecting the distance and shape ofthe object. The sonar sensor 682 f can also guide the robotic hand 72 tomove toward the object. Each of the sonar sensors 682 a, 682 b, 682 c,682 d, 682 e, 682 f, 682 g includes ultrasonic sensors, laser, radiofrequency identification (RFID), and other suitable sensors. Inaddition, each of the sonar sensors 682 a, 682 b, 682 c, 682 d, 682 e,682 f, 682 g serves as a feedback mechanism to determine whether therobotic hand 72 continues to exert additional pressure to grab theobject at such point where there is sufficient pressure to grab and liftthe object. In addition, the sonar sensor 682 f in the palm of therobotic hand 72 provides tactile sensing function to handle a kitchentool. For example, when the robotic hand 72 grabs a knife to cut beef,the amount of pressure that the robotic hand 72 exerts on the knife andapplies to the beef, allows the tactile sensor to detect when the knifefinishes slicing the beef, i.e., when the knife has no resistance. Thedistributed pressure is not only to secure the object, but also so asnot to exert too much pressure so as to, for example, not to break anegg). Furthermore, each finger on the robotic hand 72 has a sensor onthe finger tip, as shown by the first sensor 682 a on the finger tip ofthe thumb, the second sensor 682 b on the finger tip of the indexfinger, the third sensor 682 c on the finger tip of the middle finger,the fourth sensor 682 d on the finger tip of the ring finger, and thefifth sensor 682 f on the finger tip of the pinky. Each of the sensors682 a, 682 b, 682 c, 682 d, 682 e provide sensing capability on thedistance and shape of the object, sensing capability for temperature ormoisture, as well as tactile feedback capability.

The RGB-D sensor 684 and the sonar sensor 682 f in the palm, plus thesonar sensors 682 a, 682 b, 682 c, 682 d, 682 e in the finger tip ofeach finger, provide a feedback mechanism to the robotic hand 72 as ameans to grab a non-standardized object, or a non-standardized kitchentool. The robotic hands 72 may adjust the pressure to a sufficientdegree to grab ahold of the non-standardized object. A program library690 that stores sample grabbing functions 692, 694, 696 according to aspecific time interval for which the robotic hand 72 can draw from inperforming a specific grabbing function, is illustrated in FIG. 17B.FIG. 17B is a block diagram illustrating a library database 690 ofstandardized operating movements in the standardized robotic kitchenmodule 50. Standardized operating movements, which are predefined andstored in the library database 690, include grabbing, placing, andoperating a kitchen tool or a piece of kitchen equipment.

FIG. 18A is a graphical diagram illustrating that each of the robotichands 72 is coated with a artificial human-like soft-skin glove 700. Theartificial human-like soft-skin glove 700 includes a plurality ofembedded sensors that are transparent and sufficient for the robot hands72 to perform high-level mini-manipulations. In one embodiment, thesoft-skin glove 700 includes ten or more sensors to replicate a chef'shand movements.

FIG. 18B is a block diagram illustrating robotic hands coated withartificial human-like skin gloves to execute high-levelmini-manipulations based on a library database 720 ofmini-manipulations, which have been predefined and stored in the librarydatabase 720. High-level mini-manipulations refer to a sequence ofaction primitives requiring a substantial amount of interactionmovements and interaction forces and control over the same. Threeexamples of mini-manipulations are provided, which are stored in thedatabase library 720. The first example of mini-manipulation is to usethe pair of robotic hands 72 to knead the dough 722. The second exampleof mini-manipulation is to use the pair of robotic hands 72 to makeravioli 724. The third example of mini-manipulation is to use the pairof robotic hands 72 to make sushi. Each of the three examples ofmini-manipulations have a time duration and speed curve which aretracked by the computer 16.

FIG. 18C is a graphical diagram illustrating three types of taxonomy ofmanipulation actions for food preparation with continuous trajectory ofthe robotic arm 70 and the robotic hand 72 motions and forces thatresult in a desired goal state. The robotic arm 70 and the robotic hand72 execute rigid grasping and transfer 730 movements for picking up anobject with an immovable grasp and transferring them to a goal locationwithout the need for a forceful interaction. Examples of a rigidgrasping and transfer include putting the pan on the stove, picking upthe salt shaker, shaking salt into the dish, dropping ingredients into abowl, pouring the contents out of a container, tossing a salad, andflipping a pancake. The robotic arm 70 and the robotic hand 72 execute arigid grasp with forceful interaction 732 where there is a forcefulcontact between two surfaces or objects. Examples of a rigid grasp withforceful interaction include stirring a pot, opening a box, and turninga pan, and sweeping items from a cutting board into a pan. The roboticarm 70 and the robotic hand 72 execute a forceful interaction withdeformation 734 where there is a forceful contact between two surfacesor objects that results in the deformation of one of two surfaces, suchas cutting a carrot, breaking an egg, or rolling dough. For additionalinformation on the function of the human hand, deformation of the humanpalm, and its function in grasping, see the material from I. A.Kapandji, “The Physiology of the Joints, Volume 1: Upper Limb, 6e,”Churchill Livingstone, 6 edition, 2007, which this reference isincorporated by reference herein in its entirety.

FIG. 18D is a simplified flow diagram illustrating one embodiment ontaxonomy of manipulation actions for food preparation in kneading dough740. Kneading dough 740 may be a mini-manipulation that has beenpreviously predefined in the library database of mini-manipulations. Theprocess of kneading dough 740 comprises a sequence of actions (or shortmini-manipulations), including grasping the dough 742, placing the doughon a surface 744, and repeating the kneading action until one obtains adesired shape 746.

FIG. 18E is a block diagram illustrating one example of the interplayand interactions between the robotic arm 70 and the robotic hand 72. Acompliant robotic arm 750 provides a smaller payload, higher safety,more gentle actions, but less precision. An anthropomorphic robotic hand752 provides more dexterity, capable of handling human tools, is easierto retarget for a human hand motion, more compliant, but the designrequires more complexity, increase in weight, and higher product cost. Asimple robotic hand 754 is lighter in weight, less expensive, with lowerdexterity, and not able to directly use human tools. An industrialrobotic arm 756 is more precise, with higher payload capacity butgenerally not considered safe around humans and can potentially exert alarge amount of force and cause harm. One embodiment of the standardizedrobotic kitchen 50 is to utilize a first combination of the compliantarm 750 with the anthropomorphic hand 752. The other three combinationsare generally less desirable for implementation of the presentinvention.

FIG. 18F is a block diagram illustrating the robotic hand 72 using thestandardized kitchen handle 580 to attach to a custom cookware head andthe robotic arm 70 affixable to kitchen ware. In one technique to grab akitchen ware, the robotic hand 72 grabs the standardized kitchen tool580 for attaching to any one of the custom cookware heads from theillustrated choices of 760 a, 760 b, 760 c, 760 d, 760 e, and others.For example, the standardized kitchen handle 580 is attached to thecustom spatula head 760 e for use to stir-fry the ingredients in a pan.In one embodiment, the standardized kitchen handle 580 can be held bythe robotic hand 72 in just one position, which minimizes the potentialconfusion in different ways to hold the standardized kitchen handle 580.In another technique to grab a kitchen ware, the robotic arm has one ormore holders 762 that are affixable to a kitchen ware 762, where therobotic arm 70 is able to exert more forces if necessary in pressing thekitchen ware 762 during the robotic hand motion.

FIG. 19 is a block diagram illustrating an example of a database librarystructure 770 of a mini-manipulation that results in “cracking an eggwith a knife.” The mini-manipulation 770 of cracking an egg includes:how to hold an egg in the right position 772, how to hold a kniferelative to the egg 774, what is the best angle to strike the egg withthe knife 776, and how to open the cracked egg 778. Various possibleparameters for each 772, 774, 776, and 778, are tested to find the bestway to execute a specific movement. For example in holding an egg 772,the different positions, orientations, and ways to hold an egg aretested to find an optimal way to hold the egg. Second, the robotic hand72 picks up the knife from a predetermined location. The holding theknife 774 is explored as to the different positions, orientations, andthe way to hold the knife in order to find an optimal way to handle theknife. Third, the striking the egg with knife 776 is also tested for thevarious combinations of striking the knife on the egg to find the bestway to strike the egg with the knife. Consequently, the optimal way toexecute the mini-manipulation of cracking an egg with a knife 770 isstored in the library database of mini-manipulations. The savedmini-manipulation of cracking an egg with a knife 770 would comprise thebest way to hold the egg 772, the best way to hold the knife 774, andthe best way to strike the knife with the egg 776.

To create the mini-manipulation that results in cracking an egg with aknife, multiple parameter combinations must be tested to identify a setof parameters that ensure the desired functional result—that the egg iscracked—is achieved. In this example, parameters are identified todetermine how to grasp and hold an egg in such a way so as not to crushit. An appropriate knife is selected through testing, and suitableplacements are found for the fingers and palm so that it may be held forstriking. A striking motion is identified that will successfully crackan egg. An opening motion and/or force are identified that allows acracked egg to be opened successfully.

The teaching/learning process for the robotic apparatus involvesmultiple and repetitive tests to identify the necessary parameters toachieve the desired final functional result.

These tests may be performed over varying scenarios. For example, thesize of the egg can vary. The location at which it is to be cracked canvary. The knife may be at different locations. The mini-manipulationmust be successful in all of these variable circumstances.

Once the learning process has been completed, results are stored as acollection of action primitives that together are known to accomplishthe desired functional result.

FIG. 20 is a block diagram illustrating an example of recipe execution800 for a mini-manipulation with real-time adjustment. In recipeexecution 800, the robotic hands 72 execute the mini-manipulation 770 ofcracking an egg with a knife, where the optimal way to execute eachmovement in the cracking an egg operation 772, the holding a knifeoperation 774, the striking the egg with a knife operation 776, andopening the cracked egg operation 778 is selected from themini-manipulation library database. The process of executing the optimalway to carry out each of the movements 772, 774, 776, 778 ensures thatthe mini-manipulation 770 will achieve the same (or guarantee of), orsubstantially the same, outcome for that specific mini-manipulation. Themultimodal three-dimensional sensor 20 provides real-time adjustmentcapabilities 112 as to the possible variations in one or moreingredients, such as the dimension and weight of an egg.

As an example of the operative relationship between the creation of amini-manipulation in FIG. 19 and the execution of the mini-manipulationin FIG. 20, specific variables associated with the mini-manipulation of“cracking an egg with a knife,” includes an initial xyz coordinates ofegg, an initial orientation of the egg, the size of the egg, the shapeof the egg, an initial xyz coordinate of the knife, an initialorientation of the knife, the xyz coordinates where to crack the egg,speed, and the time duration of the mini-manipulation. The identifiedvariables of the mini-manipulation, “crack an egg with a knife,” arethus defined during the creation phase, where these identifiablevariables may be adjusted by the robotic food preparation engine 56during the execution phase of the associated mini-manipulation.

FIG. 21 is a flow diagram illustrating the software process 810 tocapture a chef's food preparation movements in a standardized kitchenmodule to produce the software recipe files 46 from the chef studio 44.In the chef studio 44, at step 812, the chef 49 designs the differentcomponents of a food recipe. At step 814, the robotic cooking engine 56is configured to receive the name, ID ingredient, and measurement inputsfor the recipe design that the chef 49 has selected. At step 816, thechef 49 moves food/ingredients into designated standardized cookingware/appliances and into their designated positions. For example, thechef 49 may pick two medium shallots and two medium garlic cloves, placeeight crimini mushrooms on the chopping counter, and move two 20 cm×30cm puff pastry units thawed from freezer lock F02 to a refrigerator(fridge). At step 818, the chef 49 wears the capturing gloves 26 or thehaptic costume 622, which has sensors that capture the chef's movementdata for transmission to the computer 16. At step 820, the chef 49starts working the recipe that he or she selects from step 122. At step822, the chef movement recording module 98 is configured to capture andrecord the chef's precise movements, including measurements of thechef's arms and fingers' force, pressure, and XYZ positions andorientations in real time in the standardized robotic kitchen 50. Inaddition to capturing the chef's movements, pressure, and positions, thechef movement recording module 98 is configured to record video (ofdish, ingredients, process, and interaction images) and sound (humanvoice, frying hiss, etc.) during the entire food preparation process fora particular recipe. At step 824, the robotic cooking engine 56 isconfigured to store the captured data from step 822, which includes thechef's movements from the sensors on the capturing gloves 26 and themultimodal three-dimensional sensors 30. At step 826, the recipeabstraction software module 104 is configured to generate a recipescript suitable for machine implementation. At step 828, after therecipe data has been generated and saved, the software recipe file 46 ismade available for sale or subscription to users via an app store ormarketplace to a user's computer located at home or in a restaurant, aswell as integrating the robotic cooking receipt app on a mobile device.

FIG. 22 is a flow diagram 830 illustrating the software process for foodpreparation by a robotic apparatus in the robotic standardized kitchenwith the robotic apparatus based one or more of the software recipefiles 22 received from chef studio system 44. At step 832, the user 24through the computer 15 selects a recipe bought or subscribed to fromthe chef studio 44. At step 834, the robot food preparation engine 56 inthe household robotic kitchen 48 is configured to receive inputs fromthe input module 50 for the selected recipe to be prepared. At step 836,the robot food preparation engine 56 in the household robotic kitchen 48is configured to upload the selected recipe into the memory module 102with software recipe files 46. At step 838, the robot food preparationengine 56 in the household robotic kitchen 48 is configured to calculatethe ingredient availability to complete the selected recipe and theapproximate cooking time required to finish the dish. At step 840, therobot food preparation engine 56 in the household robotic kitchen 48 isconfigured to analyze the prerequisites for the selected recipe anddecides whether or not there is any shortage or lack of ingredients, orinsufficient time to serve the dish according to the selected recipe andserving schedule. If the prerequisites are not met, at step 842, therobot food preparation engine 56 in the household robotic kitchen 48sends an alert, indicating that the ingredients should be added to ashopping list, or offers an alternate recipe or serving schedules.However, if the prerequisites are met, the robot food preparation engine56 is configured to confirm the recipe selection at step 844. At step846, after the recipe selection has been confirmed, the user 60 throughthe computer 16 moves the food/ingredients to specific standardizedcontainers and into the required positions. After the ingredients havebeen placed in the designated containers and the positions asidentified, the robot food preparation engine 56 in the householdrobotic kitchen 48 is configured to check if the start time has beentriggered at step 848. At this juncture, the household robot foodpreparation engine 56 offers a second process check to ensure that allthe prerequisites are being met. If the robot food preparation engine 56in the household robotic kitchen 48 is not ready to start the cookingprocess, the household robot food preparation engine 56 continues tocheck the prerequisites at step 850 until the start time has beentriggered. If the robot food preparation engine 56 is ready to start thecooking process, at step 852, the quality check for raw food module 96in the robot food preparation engine 56 is configured to process theprerequisites for the selected recipe and inspects each ingredient itemagainst the description of the recipe (e.g. one center-cut beeftenderloin roast) and condition (e.g. expiration/purchase date, odor,color, texture, etc.). At step 854, the robot food preparation engine 56sets the time at a “0” stage and uploads the software recipe file 46 tothe one or more robotic arms 70 and the robotic hands 72 for replicatingthe chef's cooking movements to produce a selected dish according to thesoftware recipe file 46. At step 856, the one or more robotic arms 72and hands 74 process ingredients and execute the cookingmethod/technique with identical movements as that of the chef's 49 arms,hands and fingers, with the exact pressure, the precise force, and thesame XYZ position, at the same time increments as captured and recordedfrom the chef's movements. During this time, the one or more roboticarms 70 and hands 72 compare the results of cooking against thecontrolled data (such as temperature, weight, loss, etc.) and the mediadata (such as color, appearance, smell, portion-size, etc.), asillustrated in step 858. After the data has been compared, the roboticapparatus (including the robotic arms 70 and the robotic hands 72)aligns and adjusts the results at step 860. At step 862, the robot foodpreparation engine 56 is configured to instruct the robotic apparatus tomove the completed dish to the designated serving dishes and placing thesame on the counter.

FIG. 23 is a flow diagram illustrating one embodiment of the softwareprocess for creating, testing, and validating, and storing the variousparameter combinations for a mini-manipulation library database 870. Themini-manipulation library database 870 involves a one-time success testprocess 870 (e.g., holding an egg), which is stored in a temporarylibrary, and testing the combination of one-time test results 890 (e.g.,the entire movements of cracking an egg) in the mini-manipulationdatabase library. At step 872, the computer 16 creates a newmini-manipulation (e.g., crack an egg) with a plurality of actionprimitives (or a plurality of discrete recipe actions). At step 874, thenumber of objects (e.g., an egg and a knife) associated with the newmini-manipulation are identified. The computer 16 identifies a number ofdiscrete actions or movements at step 876. At step 878, the computerselects a full possible range of key parameters (such as the positionsof an object, the orientations of the object, pressure and speed)associated with the particular new mini-manipulation. At step 880, foreach key parameter, the computer 16 tests and validates each value ofthe key parameters with all possible combinations with other keyparameters (e.g., holding an egg in one position but testing otherorientations). At step 882, the computer 16 determines if the particularset of key parameter combinations produces a reliable result. Thevalidation of the result can be done by the computer 16 or a human. Ifthe determination is negative, the computer 16 proceeds to step 886 tofind if there are other key parameter combinations that have yet to betested. At step 888, the computer 16 increments a key parameter by onein formulating the next parameter combination for further testing andevaluation for the next parameter combination. If the determination atstep 882 is positive, the computer 16 then stores the set of successfulkey parameter combinations in a temporary location library. Thetemporary location library stores one or more sets of successful keyparameter combinations (that either have the most successful test orhave the least failed results).

At step 892, the computer 16 tests and validates the specific successfulparameter combination for X number of times (such as one hundred times).At step 894, the computer 16 computes the number of failed resultsduring the repeated test of the specific successful parametercombination. At step 896, the computer 16 selects the next one-timesuccessful parameter combination from the temporary library, and returnsthe process back to step 892 for testing the next one-time successfulparameter combination X number of times. If no further one-timesuccessful parameter combination remains, the computer 16 stores thetest results of one or more sets of parameter combinations that producea reliable (or guaranteed) result at step 898. If there are more thanone reliable sets of parameter combinations, at step 899, the computer16 determines the best or optimal set of parameter combinations andstores the optimal set of parameter combination which is associated withthe specific mini-manipulation for use in the mini-manipulation librarydatabase by the robotic apparatus in the standardized robotic kitchen 50during the food preparation stages of a recipe.

FIG. 24 is a flow diagram illustrating one embodiment of the softwareprocess 900 for creating the tasks for a mini-manipulation. At step 902,the computer 16 defines a specific robotic task (e.g. cracking an eggwith a knife) with a robotic mini hand manipulator to be stored in adatabase library. The computer at step 904 identifies all differentpossible orientations of an object in each mini step (e.g. orientationof an egg and holding the egg) and at step 906 identifies all differentpositional points to hold a kitchen tool against the object (e.g.holding the knife against the egg). At step 908 the computer empiricallyidentifies all possible ways to hold an egg and to break the egg withthe knife with the right (cutting) movement profile, pressure and speed.At step 910, the computer 16 defines the various combinations to holdthe egg and positioning of the knife against the egg in order toproperly break the egg. For example, finding the combination of optimalparameters such as orientation, position, pressure and speed of theobject(s). At step 912, the computer 16 conducts a training and testingprocess to verify the reliability of various combinations, such astesting all the variations, variances, and repeats the process X timesuntil the reliability is certain for each mini-manipulation. When thechef 49 is performing certain food preparation task, (e.g. cracking anegg with a knife) the task is translated to several steps/tasks ofmini-hand manipulation to perform as part of the task at step 914. Atstep 916, the computer 16 stores the various combinations ofmini-manipulations for that specific task in the database library. Atstep 918, the computer 16 determines whether there are additional tasksto be defined and performed for any mini-manipulations. The processreturns to step 902 if there are any additional mini-manipulations to bedefined. Different embodiments of the kitchen module are possible,including a standalone kitchen module and an integrated kitchen module.The integrated kitchen module is fitted into a conventional kitchen areaof a typical house. The kitchen module operates in at least two modes, arobotic mode and a normal (manual) mode. Cracking an egg is one exampleof a mini-manipulation. The mini-manipulation library database wouldalso apply to a wide a variety of tasks, such as using a fork to grab aslab of beef by applying the right pressure in the right direction andto the proper depth to the shape and depth of the meat. At step 919, thecomputer combines the database library of predefined kitchen tasks,where each predefined kitchen task comprises one or moremini-manipulations.

FIG. 25 is a flow diagram illustrating the process 920 of assigning andutilizing a library of standardized kitchen tools, standardized objects,standardized equipment in a standardized robotic kitchen. At step 922,the computer 16 assigns each kitchen tool, object, or equipment/utensilwith a code (or bar code) that predefines the parameters of the tool,object, or equipment such as its three-dimensional position coordinatesand orientation. This process standardizes the various elements in thestandardized robotic kitchen 50, including but not limited to:standardized kitchen equipment, standardized kitchen tools, standardizedknifes, standardized forks, standardized containers, standardized pans,standardized appliances, standardized working spaces, standardizedattachments, and other standardized elements. When executing the processsteps in a cooking recipe, at step 924, the robotic cooking engine isconfigured to direct one or more robotic hands to retrieve a kitchentool, an object, a piece of equipment, a utensil, or an appliance whenprompted to access that particular kitchen tool, object, equipment,utensil or appliance, according to the food preparation process for aspecific recipe.

FIG. 26 is a flow diagram illustrating the process 926 of identifying anon-standard object through three-dimensional modeling and reasoning. Atstep 928, the computer 16 detects a non-standard object by a sensor,such as an ingredient that may have a different size, differentdimensions, and/or different weight. At step 930, the computer 16identifies the non-standard object with three-dimensional modelingsensors 66 to capture shape, dimensions, orientation and positioninformation and robotic hands 72 make a real-time adjustment to performthe appropriate food preparation tasks (e.g. cutting or picking up apiece of steak).

FIG. 27 is a flow diagram illustrating the process 932 for testing andlearning of mini-manipulations. At step 934, the computer performs afood preparation task composition analysis in which each cookingoperation (e.g. cracking an egg with a knife) is analyzed, decomposed,and constructed into a sequence of action primitives ormini-manipulations. In one embodiment, a mini-manipulation refers to asequence of one or more action primitives that accomplish a basicfunctional outcome (e.g., the egg has been cracked, or a vegetablesliced) that advances toward a specific result in preparing a food dish.In this embodiment, a mini-manipulation can be further described as alow-level mini-manipulation or a high-level mini-manipulation where alow-level mini-manipulation refers to a sequence of action primitivesthat requires minimal interaction forces and relies almost exclusivelyon the use of the robotic apparatus, and a high-level mini-manipulationrefers to a sequence of action primitives requiring a substantial amountof interaction and interaction forces and control thereof. The processloop 936 focuses on mini-manipulation and learning steps and consists oftests which are repeated many times (e.g. 100 times) to ensure thereliability of mini-manipulations. At step 938, the robotic foodpreparation engine 56 is configured to assess the knowledge of allpossibilities to perform a food preparation stage or amini-manipulation, where each mini-manipulation is tested with respectto orientations, positions/velocities, angles, forces, pressures, andspeeds with a particular mini-manipulation. A mini-manipulation or anaction primitive may involve the robotic hand 72 and a standard object,or the robotic hand 72 and a nonstandard object. At step 940, therobotic food preparation engine 56 is configured to execute themini-manipulation and determine if the outcome can be deemed successfulor a failure. At step 942, the computer 16 conducts an automatedanalysis and reasoning about the failure of the mini-manipulation. Forexample, the multimodal sensors may provide sensing feedback data on thesuccess or failure of the mini-manipulation. At step 944, the computer16 is configured to make a real-time adjustment and adjusts theparameters of the mini-manipulation execution process. At step 946, thecomputer 16 adds new information about the success or failure of theparameter adjustment to the mini-manipulation library as a learningmechanism to the robotic food preparation engine 56.

FIG. 28 is a flow diagram illustrating the process 950 for qualitycontrol and alignment functions for robotic arms. At step 952, therobotic food preparation engine 56 loads a human chef replicationsoftware recipe file 46 via the input module 50. For example, thesoftware recipe file 46 to replicate food preparation from Michelinstarred chef Arnd Beuchel's “Wiener Schnitzel”. At step 954, the roboticapparatus executes tasks with identical movements such as those for thetorso, hands, fingers, with identical pressure, force and xyz position,at an identical pace as the recorded recipe data stored based on theactions of the human chef preparing the same recipe in a standardizedkitchen module with standardized equipment based on the storedreceipt-script including all movement/motion replication data. At step956, the computer 16 monitors the food preparation process via amultimodal sensor that generates raw data supplied to abstractionsoftware where the robotic apparatus compares real-world output againstcontrolled data based on multimodal sensory data (visual, audio, and anyother sensory feedback). At step 958, the computer 16 determines ifthere any differences between the controlled data and the multimodalsensory data. At step 960, the computer 16 analyzes whether themultimodal sensory data deviates from the controlled data. If there is adeviation, at step 962, the computer 16 makes an adjustment tore-calibrate the robotic arm 70, the robotic hand 72, or other elements.At step 964, the robotic food preparation engine 16 is configured tolearn in process 964 by adding the adjustment made to one or moreparameter values to the knowledge database. At step 968, the computer 16stores the updated revision information to the knowledge databasepertaining to the corrected process, condition and parameters. If thereis no difference in deviation from step 958, the process 950 goesdirectly to step 969 in completing the execution.

FIG. 29 is a table illustrating one embodiment of a database librarystructure 970 of mini-manipulation objects for use in the standardizedrobotic kitchen. The database library structure 970 shows several fieldsfor entering and storing information for a particular mini-manipulation,including (1) the name of the mini-manipulation, (2) the assigned codeof the mini-manipulation, (3) the code(s) of standardized equipment andtools associated with the performance of the mini-manipulation, (4) theinitial position and orientation of the manipulated (standard ornon-standard) objects (ingredients and tools), (5) parameters/variablesdefined by the user (or extracted from the recorded recipe duringexecution), (6) sequence of robotic hand movements (control signals forall servos) and connecting feedback parameters (from any sensor or videomonitoring system) of mini-manipulations on the timeline. The parametersfor a particular mini-manipulation may differ depending on thecomplexity and objects that are necessary to perform themini-manipulation. In this example, four parameters are identified: thestarting XYZ position coordinates in the volume of the standardizedkitchen module, the speed, the object size, and the object shape. Boththe object size and the object shape may be defined or described bynon-standard parameters.

FIG. 30 is a table illustrating a database library structure 972 ofstandardized objects for use in the standardized robotic kitchen. Thestandard object database library structure 972 shows several fields tostore information pertaining to a standard object, including (1) thename of an object, (2) an image of the object, (3) an assigned code forthe object, (4) a virtual 3D model with full dimensions of the object inan XYZ coordinate-matrix with the preferred resolution predefined, (5) avirtual vector model of the object (if available), (6) definition andmarking of the working elements of the object (the elements, which maybe in contact with hands and other objects for manipulation), and (7) aninitial standard orientation of the object for each specificmanipulation.

FIG. 32 depicts the execution of process 1000 used to check for thequality of the ingredients to be used as part of the recipe replicationprocess by the standardized robotic kitchen. The multi-modal sensorsystem video-sensing element is able to implement process 1006, whichuses color-detection and spectral analysis to detect discolorationindicating possible spoilage. Similarly using an ammonia-sensitivesensor system, whether embedded in the kitchen or part of a mobile probehandled by the robotic hands, further potential for spoilage can bedetected. Additional haptic sensors in the robotic hands and fingerswould allow for validating the freshness of the ingredient through thetouch-0sensing process 1004, where the firmness and resistance tocontact forces is measured (amount and rate of deflection as a functionof compression-distance). As an example, for fish the color (deep red)and moisture content of the gills is an indicator of freshness, as theeyes which should be clear (not fogged), and the proper temperature ofthe flesh of a properly thawed fish should not exceed 40 deg F.Additional contact-sensors on the finger-tips are able to carry outadditional quality check 1002 related to the temperature, texture andoverall weight of the ingredient through touching, rubbing andholding/pickup motions. All the data collected through these hapticsensors and video-imagery can be used in a processing algorithm todecide on the freshness of the ingredient and make decisions on whetherto use it or dispose of it.

FIG. 32 depicts the robotic recipe-script replication process 1010,wherein a multi-modal sensor outfitted head 20, and dual arms withmulti-fingered hands 72 holding ingredients and utensils, interact withcookware 1012. The robotic sensor head 20 with a multi-modal sensor unitis used to continually model and monitor the three-dimensionaltask-space being worked by both robotic arms while also providing datato the task-abstraction module to identify tools and utensils,appliances and their contents and variables, so as to allow them to becompared to the cooking-process sequence generated recipe-steps toensure the execution is proceeding along the computer-storedsequence-data for the recipe. Additional sensors in the robotic sensorhead 20 are used in the audible domain to listen and smell duringsignificant parts of the cooking process. The robotic hands 72 and theirhaptic sensors are used to properly handle respective ingredients, suchas an egg in this case; the sensors in the fingers and palm are able tofor example detect a usable egg by way of surface texture and weight andits distribution and hold and orient the egg without breaking it. Themulti-fingered robotic hands 72 are also capable of fetching andhandling particular cookware, such as a bowl in this case, and grab andhandle cooking utensils (a whisk in this case), with proper motions andforce application so as to properly process food ingredients (e.g.cracking an egg, separating the yolks and beating the egg-white until astiff composition is achieved) as specified in the recipe-script.

FIG. 33 depicts the ingredient storage system notion 1020, wherein foodstorage containers 1022, capable of storing any of the needed cookingingredients (e.g. meats, fish, poultry, shellfish, vegetables, etc.),are outfitted with sensors to measure and monitor the freshness of therespective ingredient. The monitoring sensors embedded in the foodstorage containers 1022 include, but are not limited to, ammonia sensors1030, volatile organic compound sensors 1032, internal containertemperature sensors 1026 and humidity sensors 1028. Additionally amanual probe can be used, whether employed by the human chef or therobotic arms and hands, to allow for key measurements (such astemperature) within a volume of a larger ingredient (e.g. internal meattemperature).

FIG. 34 depicts the measurement and analysis process 1040 carried out aspart of the freshness and quality check for ingredients placed in foodstorage containers 1042 containing sensors and detection devices (e.g. atemperature probe/needle). A container is able to forward its data setby way of a metadata tag 1044, specifying its container-ID, andincluding the temperature data 1046, humidity data 1048, ammonia leveldata 1050, volatile organic compound data 1052 over a wirelessdata-network through a communication step 1056, to a main server where afood control quality engine processes the container data. The processingstep 1060 uses the container-specific data 1044 and compares it todata-values and -ranges considered acceptable, which are stored andretrieved from media 1058 by a data retrieval and storage process 1054.A set of algorithms then make the decision as to the suitability of theingredient, providing a real-time food quality analysis result over thedata-network via a separate communication process 1062. The qualityanalysis results are then utilized in another process 1064, where theresults are forwarded to the robotic arms for further action and mayalso be displayed remotely on a screen (such as a smartphone or otherdisplay) for a user to decide if the ingredient is to be used in thecooking process for later consumption or disposed of as spoiled.

FIG. 35 depicts the functionalities and process-steps of pre-filledingredient containers 1070 when used in the standardized kitchen,whether it be the standardized robotic kitchen or the chef studio.Ingredient containers 1070 are designed in different sizes 1082 andvaried usages in mind and are suitable for proper storage environments1080 to accommodate perishable items by way of refrigeration, freezing,chilling, etc. to achieve specific storage temperature ranges.Additionally, ingredient storage containers 1070 are also designed tosuit different types of ingredients 1072, with containers alreadypre-labeled and pre-filled with solid (salt, flour, rice, etc.),viscous/pasty (mustard, mayonnaise, marzipan, jams, etc.) or liquid(water, oil, milk, juice, etc.) ingredients, where dispensing processes1074 utilize a variety of different application devices (dropper, chute,peristaltic dosing pump, etc.) depending on the ingredient type, withexact computer-controllable dispensing by way of a dosage control engine1084 running a dosage control process 1076 ensuring that the properamount of ingredient is dispensed at the right time. It should be notedthat the recipe-specified dosage is adjustable to suit personal tastesor diets (low sodium, etc.), by way of a menu-interface or even througha remote phone application. The dosage determination process 1078 iscarried out by the dosage control engine 1084, based on the amountspecified in the recipe, with dispensing occurring either through manualrelease command or remote computer control based on the detection of aparticular dispensing container at the exit point of the dispenser.

FIG. 36 is a block diagram illustrating a recipe system structure 1000for use in the standardized robotic kitchen 50. The food preparationprocess 1100 is shown as divided into multiple stages along the cookingtimeline, with each stage having or more raw data blocks for each stage1102, stage 1004, stage 1106 and stage 1108. The data blocks can containsuch elements as video-imagery, audio-recordings, textual descriptions,as well as the machine-readable and -understandable set of instructionsand commands that form a part of the control program. The raw data setis contained within the recipe structure and representative of eachcooking stage along a timeline divided into many time-sequenced stages,with varying levels of time-intervals and -sequences, all the way fromthe start of the recipe replication process to the end of the cookingprocess, or any sub-process therein.

FIGS. 37A-C are block diagrams illustrating recipe search menus for usein the standardized robotic kitchen. As shown in FIG. 37A, a recipesearch menu 1120 provides most popular categories such as type ofcuisine (e.g. Italian, French, Chinese), the basis of ingredients of thedish (e.g. fish, pork, beef, pasta), or criteria and range such ascooking time range (e.g. less than 60 minutes, between 20 to 40 minutes)as well as conducting a keyword search (e.g. ricotta cavatelli,migliaccio cake). A selected personalized recipe may excluding a recipewith allergic ingredients in which a user can indicate allergicingredients that the user may refrain from in a personal user profile.In FIG. 37B, the user may select a search criteria, including therequirements of a cooking time less than 44 minutes, serving sufficientportions for 7 people, providing vegetarian dish options, with a totalcalories of 4521 or less. The different types of dishes 1122 are shownin FIG. 37C where menu 1120 has hierarchical levels such that the usermay select a category (e.g. type of dish) 1122, which then expands tothe next level sub-categories (e.g. appetizers, salads, entrees . . . )to refine the selections. A screen shot of an implemented recipecreation and submission is illustrated in FIG. 37D. Additional screenshots of various graphical user interface and menu options areillustrated in FIG. 37N-V.

One embodiment of the flow charts in functioning as a recipe filter, aningredient filter, an equipment filter, an account and social networkaccess, a personal partner page, a shopping cart page, and theinformation on the purchased recipe, registration setting, create arecipe are illustrated in FIG. 37E through 37M, which illustrate thevarious functions that the robotic food preparation software 14 iscapable of performing based on the filtering of databases and presentingthe information to the user. As demonstrated in FIG. 37E, a platformuser can access the recipe section and choose the desired recipe filters1130 for automatic robotic cooking. The most common filter types includetypes of cuisine (e.g. Chinese, French, Italian), type of cooking (e.g.bake, steam, fry), vegetarian dishes, and diabetic food. The user willbe able to view the recipe details, such as description, photo,ingredients, price, and ratings, from the filtered search result. InFIG. 37F, the user can choose the desired ingredient filters 1132, suchas organic, type of ingredient, or brand of ingredient, for his purpose.In FIG. 37G, the user can apply the equipment filters 1134 for theautomatic robotic kitchen modules, such as the type, the brand, and themanufacturer of equipment. After making the selections, the user will beable to purchase recipes, ingredients, or equipment product directlythrough the system portal from the associated vendors. The platformallows the users to create additional filters and parameters for his ownpurpose, which makes the entire system customizable and constantlyrenewing. The user-added filters and parameters will appear as systemfilters after approval by moderator.

In FIG. 37H, a user is able to connect to other users and vendorsthrough the platform's social professional network by logging into theuser account 1136. The identity of the network user is verified,possibly through the credit card and the address details. The accountportal also serves as a trading platform for users to share or selltheir recipes, as well as advertising to other users. The user canmanage his account finance and equipment through the account portal aswell.

An example of partnership between users of the platform is demonstratedin FIG. 371. One user can provide all the information and details forhis ingredients and another user does the same for his equipment. Allinformation must be filtered through a moderator before adding to theplatform/website database. In FIG. 37J, a user can see the informationfor his purchases in the shopping cart 1140. Other options, such asdelivery and payment method, can also be changed. The user can alsopurchase more ingredients or equipment, based on the recipes in hisshopping cart.

FIG. 37K shows the other information on the purchased recipes can beaccessed from the recipes page 1560. The user can read, hear, and watchhow to cook, as well as execute automatic robotic cooking. Communicationwith the vendors or technical support regarding the recipe is alsopossible from the recipes page.

FIG. 37L is a block diagram that illustrate the different layers of theplatform from the “My account” page 1136 and Settings page 1138. Fromthe “My account” page, the user will be able to read professionalcooking news or blogs, and can write an article to publish. Through therecipe page under “My account”, there are multiple ways a user cancreate his own recipe 1570, as shown in FIG. 37M. The user can create arecipe by creating an automatic robotic cooking script either bycapturing chief cooking movements or by choosing manipulation sequencesfrom software library. The user can also create recipe by simply listingthe ingredient/equipment, then add audio, video, or picture. The usercan edit all recipes from the recipe page.

FIG. 38 is a block diagram illustrating a recipe search menu 1150 byselecting fields for use in the standardized robotic kitchen. Byselecting a category with a search criteria or range, the user 60receives a return page that lists the various recipes results. The user60 is able to sort the results by criteria such as a user rating (e.g.from high to low), an expert rating (e.g. from high to low), or theduration of the food preparation (e.g. from shorter to longer). Thecomputer display may contain a photo/media, title, description, ratingsand price information of the recipe, with an optional tab of the “readmore” button that brings up a complete recipe page for browsing furtherinformation about the recipe.

The standardized robotic kitchen 50 in FIG. 39 depicts a possibleconfiguration for the use of an augmented sensor system 1854. Theaugmented sensor system 1854 shows a single augmented sensor system 1854placed on a movable computer-controllable linear rail travelling thelength of the kitchen axis with the intent to effectively cover thecomplete visible three-dimensional workspace of the standardizedkitchen.

Based on the proper placement of the augmented sensor system 1854 placedsomewhere in the robotic kitchen, such as on a computer-controllablerailing, or on the torso of a robot with arms and hands, allows for3D-tracking and raw data generation, both during chef-monitoring formachine-specific recipe-script generation, and monitoring the progressand successful completion of the robotically-executed steps in thestages of the dish replication in the standardized robotic kitchen 50.

The standardized robotic kitchen 50 in FIG. 39 depicts a possibleconfiguration for the use of an augmented sensor system 20. Thestandardized robotic kitchen 50 shows a single augmented sensor system20 placed on a movable computer-controllable linear rail travelling thelength of the kitchen axis with the intent to effectively cover thecomplete visible three-dimensional workspace of the standardizedkitchen.

FIG. 40. is a block diagram illustrating the standardized kitchen module50 with multiple camera sensors and/or lasers 20 for real-timethree-dimensional modeling 1160 of the food preparation environment. Therobotic kitchen cooking system 48 includes a three-dimensionalelectronic sensor that is capable of providing real-time raw data for acomputer to create a three-dimensional model of the kitchen operatingenvironment. One possible implementation of the real-timethree-dimensional modeling process involves the use of three-dimensionallaser scanning. An alternative implementation of the real-timethree-dimensional modeling is to use one or more video cameras. Yet athird method involves the use of a projected light-pattern observed by acamera, so-called structured-light imaging. The three-dimensionalelectronic sensor scans the kitchen operating environment in real-timeto provide a visual representation (shape and dimensional data) 1162 ofthe working space in the kitchen module. For example, thethree-dimensional electronic sensor captures in real-time thethree-dimensional images of whether the robotic arm/hand has picked upmeat or fish. The three-dimensional model of the kitchen also serves assort of a ‘human-eye’ for making adjustments to grab an object, as someobjects may have nonstandard dimensions. The compute processing system16 generates a computer model of the three-dimensional geometry andobjects in the workspace and provides controls signals 1164 back to thestandardized robotic kitchen 50. For instance, three-dimensionalmodeling of the kitchen can provide a three-dimensional resolution gridwith a desirable spacing, such as with 1 centimeter spacing between thegrid points.

The standardized robotic kitchen 50 depicts another possibleconfiguration for the use of one or more augmented sensor systems 20.The standardized robotic kitchen 50 shows a multitude of augmentedsensor systems 20 placed in the corners above the kitchen work-surfacealong the length of the kitchen axis with the intent to effectivelycover the complete visible three-dimensional workspace of thestandardized robotic kitchen 50.

The proper placement of the augmented sensor system 20 in thestandardized robotic kitchen 50, allows for three-dimensional sensing,using video-cameras, lasers, sonars and other two- and three-dimensionalsensor systems to enable the collection of raw data to assist in thecreation of processed data for real-time dynamic models of shape,location, orientation and activity for robotic arms, hands, tools,equipment and appliances, as they relate to the different steps in themultiple sequential stages of dish replication in the standardizedrobotic kitchen 50.

Raw data is collected at each point in time to allow the raw data to beprocessed to be able to extract the shape, dimension, location andorientation of all objects of importance to the different steps in themultiple sequential stages of dish replication in the standardizedrobotic kitchen 50 in a step 1162. The processed data is furtheranalyzed by the computer system to allow the controller of thestandardized robotic kitchen to adjust robotic arm and hand trajectoriesand mini-manipulations, by modifying the control signals defined by therobotic script. Adaptations to the recipe-script execution and thuscontrol signals is essential in successfully completing each stage ofthe replication for a particular dish, given the potential forvariability for many variables (ingredients, temperature, etc.). Theprocess of recipe-script execution based on key measurable variables isan essential part of the use of the augmented (also termed multi-modal)sensor system 20 during the execution of the replicating steps for aparticular dish in a standardized robotic kitchen 50.

FIG. 41A is a diagram illustrating a robotic kitchen prototype. Theprototype kitchen consists of three levels, the top level includes arail system for two arms to move along when cooking, an extractible hoodfor two robot arms to return to a charging dock and allow them to bestored when not used for cooking or when the kitchen is set to manualcooking mode. The mid level includes sinks, stove, griller, oven, and aworking counter top with access to ingredients storage. The middle levelhas also a computer monitor to operate the equipment, choose the recipe,watching the video and text instructions, and listening to the audioinstruction. The lower level includes an automatic container system tostore food/ingredients at their best conditions, with the possibility toautomatically deliver ingredients to the cooking volume as required bythe recipe. The kitchen prototype also includes an oven, dishwasher,cooking tools, accessories, cookware organizer, drawers and recycle bin.

FIG. 41B is a diagram illustrating a robotic kitchen prototype with atransparent material enclosure that serves as a protection mechanismwhile the robotic cooking process is occurring to prevent causingpotential injuries to surrounding humans. The transparent materialenclosure can be made from a variety of transparent materials, such asglass, fiberglass, plastics, or any other suitable material. In oneexample, the transparent material enclosure comprises an automatic glassdoor (or doors). As shown in this embodiment, the automatic glass doorsare positioned to slide up-down or down-up (from bottom section) toclose for safety reasons during the cooking process involving the use ofrobotic arms. A variation in the design of the transparent materialenclosure is possible, such as vertically sliding down, verticallysliding up, horizontally from left to right, horizontally from right toleft, or any other methods that place allow for the transparent materialenclosure in the kitchen to serve as a protection mechanism.

FIG. 41C depicts an embodiment of the standardized robotic kitchen,where the volume prescribed by the countertop surface and the undersideof the hood, has horizontally sliding glass doors 1190, that can bemanually, or under computer control, moved left or right to separate theworkspace of the robotic arms/hands from its surroundings for suchpurposes as safeguarding any human standing near the kitchen, or limitcontamination into/out-of the kitchen work-area, or even allow forbetter climate control within the enclosed volume. The automatic slidingglass doors slide left-right to close for safety reasons during thecooking processes involving the use of the robotic arms.

FIG. 41D depicts an embodiment of the standardized robotic kitchen,where the countertop or work-surface includes an area with asliding-door 1200 access to the ingredient-storage volume in the bottomcabinet volume of the robotic kitchen counter. The doors can be slidopen manually, or under computer control, to allow access to theingredient containers therein. Either manually, or under computercontrol, one or more specific containers can be fed to countertop levelby the ingredient storage-and-supply unit, allowing manual access (inthis depiction by the robotic arms/hands) to the container, its lid andthus the contents of the container. The robotic arms/hands can then openthe lid, retrieve the ingredient(s) as needed, and place theingredient(s) in the appropriate place (plate, pan, pot, etc.), beforere-sealing the container and placing it back on or into the ingredientstorage-and-supply unit. The ingredient storage-and-supply unit thenplaces the container back into the appropriate location within the unitfor later re-use, cleaning or re-stocking. This process of supplying andre-stacking ingredient containers for access by the robotic arms/handsis an integral and repeating process that forms part of therecipe-script as certain steps within the recipe replication processcall for one or more ingredients of a certain type, based on the stageof the recipe-script execution the standardized robotic kitchen 50 mightbe involved in.

To access the ingredients storage-and-supply unit, part of thecountertop with sliding doors can be opened, where the recipe softwarecontrols the doors and moves designated containers and ingredients tothe access location where the robotic arm(s) may pick up the containers,open the lid, remove the ingredients out of the containers to adesignated place, reseal the lid and move the containers back intostorage. The container is moved from the access location back to itsdefault location in the storage unit, and a new/next container item isthen uploaded to the access location to be picked up.

An alternative embodiment for an ingredient storage-and-supply unit 1210is depicted in FIG. 41E. Specific or repetitively used ingredients(salt, sugar, flour, oil, etc.) can be dispensed usingcomputer-controlled feeding mechanisms or allow for hand-triggered,whether by human or robotic hands or fingers, release of a specifiedamount of a specific ingredient. The amount of ingredient to bedispensed can be manually entered by the human or robotic hand on atouch-panel, or provided via computer-control. The dispensed ingredientcan then be collected or fed into a piece of kitchen equipment (bowl,pan, pot, etc.) at any time during the recipe replication process. Thisembodiment of an ingredient supply and dispensing system can be thoughtof as more cost- and space-efficient approach while also reducingcontainer-handling complexity as well as wasted motion-time by the robotarms/hands.

In FIG. 41F an embodiment of the standardized robotic kitchen includes abacksplash area 1220, wherein is mounted a virtual monitor/display witha touchscreen area to allow a human operating the kitchen in manual modeto interact with the robotic kitchen and its elements. Acomputer-projected image and a separate camera monitoring the projectedarea can tell where the human hand and its finger are located whenmaking a specific choice based on a location in the projected image,upon which the system then acts accordingly. The virtual touchscreenallows for access to all control and monitoring functions for allaspects of the equipment within the standardized robotic kitchen 50,retrieval and storage of recipes, reviewing stored videos of complete orpartial recipe execution steps by a human chef, as well as listening toaudible playback of the human chef voicing descriptions and instructionsrelated to a particular step or operation in a particular recipe.

FIG. 41G depicts a single or a series of robotic hard automationdevice(s) 1230 which are built into the standardized robotic kitchen.The device or devices are programmable and controllable remotely by acomputer and are designed to feed or provide pre-packaged orpre-measured amounts of dedicated ingredient elements needed in therecipe replication process, such as spices (salt, pepper, etc.), liquids(water, oil, etc.) or other dry ingredients (flour, sugar, bakingpowder, etc.). These robotic automation devices 1230 are located so asto make them readily accessible to the robotic arms/hands to allow themto be used by the robotic arms/hands or those of a human chef, to setand/or trigger the release of a determined amount of an ingredient ofchoice based on the needs specified in the recipe-script.

FIG. 41H depicts a single or a series of robotic hard automationdevice(s) 1340, which are built into the standardized robotic kitchen.The device or devices are programmable and controllable remotely by acomputer and are designed to feed or provide pre-packaged orpre-measured amounts of common and repetitively used ingredient elementsneeded in the recipe replication process, where a dosage controlengine/system, is capable of providing just the proper amount to aspecific piece of equipment, such as a bowl, pot or pan. These roboticautomation devices 1340 are located so as to make them readilyaccessible to the robotic arms/hands to allow them to be used by therobotic arms/hands or those of a human cook, to set and/or trigger therelease of a dosage-engine controlled amount of an ingredient of choicebased on the needs specified in the recipe-script. This embodiment of aningredient supply and dispensing system can be thought of as more cost-and space-efficient approach while also reducing container-handlingcomplexity as well as wasted motion-time by the robot arms/hands.

FIG. 411 depicts the standardized robotic kitchen outfitted with both aventilation system 1250 to extract fumes and steam during the automatedcooking process, as well as an automatic smoke/flame detection andsuppression system 1252 to extinguish any source of noxious smoke anddangerous fire also allowing the safety glass of the sliding doors toenclose the standardized robotic kitchen 50 to contain the affectedspace.

FIG. 41J depicts the standardized robotic kitchen 50 with a wastemanagement system 1260 which is located within a location in the lowercabinet so as to allow for easy and rapid disposal of recyclable (glass,aluminum, etc.) and non-recyclable (food scraps, etc.) items by way of aset of trash containers with removable lids, which contain sealingelements (gaskets, o-rings, etc.) to provide for an airtight seal tokeep odors from escaping into the standardized robotic kitchen 50.

FIG. 41K depicts the standardized robotic kitchen 50 with a top-loadeddishwasher 1270 located within a certain location in the kitchen forease of robotic loading and unloading. The dishwasher includes a sealinglid, which during automated recipe replication step execution can alsobe used as a cutting board or workspace with an integral drainagegroove.

FIG. 41L depicts the standardized kitchen with an instrumentedingredient quality-check system 1280 comprised of an instrumented panelwith sensors and a food-probe. The area includes sensors on thebacksplash capable of detecting multiple physical and chemicalcharacteristics of ingredients placed within the area, including but notlimited to spoilage (ammonia sensor), temperature (thermocouple),volatile organic compounds (emitted upon biomass decomposition), as wellas moisture/humidity (hygrometer) content. A food probe using atemperature-sensor (thermocouple) detection device can also be presentto be wielded by the robotic arms/hands to probe the internal propertiesof a particular cooking ingredient or element (such as internaltemperature of red meat, poultry, etc.).

FIG. 42A depicts an embodiment of a standardized robotic kitchen in planview 50, whereby it should be understood that the elements therein couldbe arranged in a different fashion. The standardized robotic kitchen isdivided in to three levels, namely the top level 1292-1, the counterlevel 1292-2 and the lower level 1292-3.

The top level 1292-1 contains multiple cabinet-type modules withdifferent units to perform specific kitchen functions by way of built-inappliances and equipment. At the simplest level a shelf/cabinet storagearea 1294 is included, a cabinet volume 1296 used for storing andaccessing cooking tools and utensils and other cooking and serving ware(cooking, baking, plating, etc.), a storage ripening cabinet volume 1298for particular ingredients (e.g. fruit and vegetables, etc.), a chilledstorage zone 1300 for such items as lettuce and onions, a frozen storagecabinet volume 1302 for deep-frozen items, and another storage pantryzone 1304 for other ingredients and rarely used spices, etc.

The counter level 1292-2 not only houses the robotic arms 70, but alsoincludes a serving counter 1306, a counter area with a sink 1308,another counter area 1310 with removable working surfaces(cutting/chopping board, etc.), a charcoal-based slatted grill 1312 anda multi-purpose area for other cooking appliances 1314, including astove, cooker, steamer and poacher.

The lower level 1292-3 houses the combination convection oven andmicrowave 1316, the dish-washer 1318 and a larger cabinet volume 1320that holds and stores additional frequently used cooking and bakingware, as well as tableware and packing materials and cutlery.

FIG. 42B depicts a perspective view 50 of the standardized robotickitchen, depicting the locations of the top level 1292-1, counter level1292-2 and the lower level 1294-3, within an xyz coordinate frame withaxes for x 1322, y 1324 and z 1326 to allow for proper geometricreferencing for positioning of the robotic arms 34 within thestandardized robotic kitchen.

The perspective view of the robotic kitchen 50 clearly identifies one ofthe many possible layouts and locations for equipment at all threelevels, including the top level 1292-1 (storage pantry 1304,standardized cooking tools and ware 1320, storage ripening zone 1298,chilled storage zone 1300, and frozen storage zone 1302, the counterlevel 1292-2 (robotic arms 70, sink 1308, chopping/cutting area 1310,charcoal grill 1312, cooking appliances 1314 and serving counter 1306)and the lower level (dish-washer 1318 and oven and microwave 1316).

FIG. 43A depicts a plan view of one possible physical embodiment of thestandardized robotic kitchen layout, where the kitchen is built into amore linear substantially rectangular horizontal layout depicting abuilt-in monitor 1328 for a user to operate the equipment, choose arecipe, watch video and listen to the recorded chef's instructions, aswell as automatically computer-controlled left/right movable transparentdoors 1330 for enclosing the open faces of the standardized roboticcooking volume during operation of the robotic arms.

FIG. 43B depicts a perspective view of one possible physical embodimentof the standardized robotic kitchen layout, where the kitchen is builtinto a more linear substantially rectangular horizontal layout depictinga built-in monitor 1332 for a user to operate the equipment, choose arecipe, watch video and listen to the recorded chef's instructions, aswell as automatically computer-controlled left/right movable transparentdoors 1334 for enclosing the open faces of the standardized roboticcooking volume during operation of the robotic arms. Sample screen shotsin the standardized robotic kitchen are illustrated in FIGS. 43C-E,while FIG. 43F depicts a sample kitchen module specification.

FIG. 44A depicts a plan view of another possible physical embodiment ofthe standardized robotic kitchen layout, where the kitchen is built intoa more linear substantially rectangular horizontal layout depicting abuilt-in monitor 1336 for a user to operate the equipment, choose arecipe, watch video and listen to the recorded chef's instructions, aswell as automatically computer-controlled up/down movable transparentdoors 1338 for enclosing the open faces of the standardized roboticcooking volume during operation of the robotic arms.

FIG. 44B depicts a perspective view of another possible physicalembodiment of the standardized robotic kitchen layout, where the kitchenis built into a more linear substantially rectangular horizontal layoutdepicting a built-in monitor 1340 for a user to operate the equipment,choose a recipe, watch video and listen to the recorded chef'sinstructions, as well as automatically computer-controlled up/downmovable transparent doors 1342 for enclosing the open faces of thestandardized robotic cooking volume during operation of the roboticarms.

FIG. 45 depicts a perspective layout view of a telescopic lift (or“telescopic actuator” or “telescopic mast”) 1350 in the standardizedrobotic kitchen 50 in which a pair of robotic arms, wrists andmulti-fingered hands move as a unit on a prismatically (through linearstaged extension) and telescopically actuated torso along the verticaly-axis 1352 and the horizontal x-axis 1354, as well as rotationallyabout the vertical y-axis running through the centerline of its owntorso. Actuators (including for example, but not limited to, telescopicactuators or telescopic masts) are embedded in the torso and upper levelto allow for these linear and rotary motions so as to allow the roboticarms to be moved to different places in the standardized robotic kitchenduring all parts of the replication of the recipe spelled out in therecipe script. These multiple motions are necessary to be able toproperly replicate the motions of a human chef 49 as observed in thechef studio kitchen setup during the creation of the dish when cooked bythe human chef.

FIG. 46A depicts a plan view of one physical embodiment 1356 of thestandardized robotic kitchen layout, where the kitchen is built into amore linear substantially rectangular horizontal layout depicting a setof dual robotic arms with wrists and multi-fingered hands, where each ofthe arm bases is mounted neither on a set of movable rails nor on arotatable torso, but rather rigidly and unmovably mounted on one and thesame of the robotic kitchen vertical surfaces, thereby defining andfixing the location and dimensions of the robotic torso, yet stillallowing both robotic arms to work collaboratively and reach all areasof the cooking surfaces and equipment.

FIG. 46B depicts a perspective view of one physical embodiment 1358 ofthe standardized robotic kitchen layout, where the kitchen is built intoa more linear substantially rectangular horizontal layout depicting aset of dual robotic arms with wrists and multi-fingered hands, whereeach of the arm bases is not mounted neither on a set of movable railsnor on a rotatable torso, but rather rigidly and unmovably mounted onone and the same of the robotic kitchen vertical surfaces, therebydefining and fixing the location and dimensions of the robotic torso,yet still allowing both robotic arms to work collaboratively and reachall areas of the cooking surfaces and equipment (oven on back wall,cooktop beneath the robotic arms and sink to one side of the roboticarms).

FIG. 46C depicts a dimensioned front view of one possible physicalembodiment 1360 of the standardized robotic kitchen, denoting its heightalong the y-axis and width along the x-axis to be 2284 mm overall.

FIG. 46D depicts a dimensioned side section view of one possiblephysical embodiment 1362 of the standardized robotic kitchen, denotingits height along the y-axis to be 2164 mm and 3415 mm, respectively.

FIG. 46E depicts a dimensioned side view of one physical embodiment 1364of the standardized robotic kitchen, denoting its height along they-axis and depth along the z-axis to be 2284 mm and 1504 mm,respectively.

FIG. 46F depicts a dimensioned top section view of one physicalembodiment 1366 of the standardized robotic kitchen, including a pair ofrobotic arms 1368, denoting the depth of the entire robotic kitchenmodule along the z-axis to be 1504 mm overall.

FIG. 46G depicts a three-view, augmented by a section-view, of onephysical embodiment of the standardized robotic kitchen, showing theoverall length along the x-axis to be 3415 mm, the overall height alongthe y-axis to be 2164 mm, and the overall depth along the z-axis to be1504 mm, where the overall height in the sectional side-view indicatesan overall height along the z-axis of 2284 mm.

FIG. 47 is a block diagram illustrating a programmable storage system 88for use with the standardized robotic kitchen 50. The programmablestorage system 88 is structured in the standardized robotic kitchen 50based on the relative xy position coordinates within the storage system88. In this example, the programmable storage system 88 has twenty seven(27; arranged in a 9×3 matrix) storage locations that have nine columnsand three rows. The programmable storage system 88 can serve as thefreezer location or the refrigeration location. In this embodiment, eachof the twenty-seven programmable storage locations includes four typesof sensors: a pressure sensor 1370, a humidity sensor 1372, atemperature sensor 1374, and a smell (olfactory) sensor 1376. With eachstorage location recognizable by its xy coordinates, the roboticapparatus is able to access a selected programmable storage location toobtain the necessary food item(s) in the location to prepare a dish. Thecomputer 16 can also monitor each programmable storage location for theproper temperature, proper humidity, proper pressure, and proper smellprofiles to ensure optimal storage conditions for particular food itemsor ingredients are monitored and maintained.

FIG. 48 depicts an elevation view of the container storage station 86,where temperature, humidity and relative oxygen content (and other roomconditions) can be monitored and controlled by a computer. Included inthis storage container unit can be, but it is not limited to, apantry/dry storage area 1304, a ripening area 1298 with separatelycontrollable temperature and humidity (for fruit/vegetables), ofimportance to wine, a chiller unit 1300 for lower temperature storagefor produce/fruit/meats so as to optimize shelf life, and a freezer unit1302 for long-term storage of other items (meats, baked goods, seafood,ice cream, etc.).

FIG. 49 depicts an elevation view of ingredient containers 1380 to beaccessed by a human chef and the robotic arms and multi-fingered hands.This section of the standardized robotic kitchen includes, but is notnecessarily limited to, multiple units including an ingredient qualitymonitoring dashboard (display) 1382, a computerized measurement unit1384, which includes a barcode scanner, camera and scale, a separatecountertop 1386 with automated rack-shelving for ingredient check-in andcheck-out, and a recycling unit 1388 for disposal of recyclable hard(glass, aluminum, metals, etc.) and soft goods (food rests and scraps,etc.) suitable for recycling.

FIG. 50 depicts the ingredient quality-monitoring dashboard 1390, whichis a computer-controlled display for use by the human chef. The displayallows the user to view multiple items of importance to theingredient-supply and ingredient-quality aspect of human and roboticcooking. These include the display of the ingredient inventory overview1392 outlining what is available, the individual ingredient selected andits nutritional content and relative distribution 1394, the amount anddedicated storage as a function of storage category 1396 (meats,vegetables, etc.), a schedule 1398 depicting pending expiry dates andfulfillment/replenishment dates and items, an area for any kinds ofalerts 1400 (sensed spoilage, abnormal temperatures or malfunctions,etc.), and the option of voice-interpreter command input 1402, to allowthe human user to interact with the computerized inventory system by wayof the dashboard 1390.

FIG. 51 is a table illustrating one example of a library database 1410of recipe parameters. The library database 1410 of recipe parametersincludes many categories: a meal grouping profile 1402, types of cuisine1404, a media library 1406, recipe data 1408, robotic kitchen tools andequipment 1410, ingredient groupings 1412, ingredient data 1414, andcooking techniques 1416. Each of these categories provides a listing ofthe detailed choices that are available in selecting a recipe. The mealgroup profile includes parameters like age, gender, weight, allergy,medication and lifestyle. The types of cuisine group profile 1404include cuisine type by region, culture, or religion, and the types ofcooking equipment group profile 1410 include items such as pan, grill,or oven and the cooking duration time. The recipe data grouping profile1408 contains such items as the recipe name, version, cooking andpreparation time, tools and appliances needed, etc. The ingredientgrouping profile 1412 contains ingredients grouped into items such asdairy products, fruit and vegetables, grains and other carbohydrates,fluids of various types, and protein of various kinds (meats, beans),etc. The ingredient data group profile 1414 contains ingredientdescriptor data such as the name, description, nutritional information,storage and handling instructions, etc. The cooking techniques groupprofile 1416 contains information on specific cooking techniques groupedinto such areas as mechanical techniques (basting, chopping, grating,mincing, etc.) and chemical processing techniques (marinating, pickling,fermenting, smoking, etc.).

FIG. 52 is a flow diagram illustrating one embodiment of the process1420 of one embodiment of recording a chef's food preparation process.At step 1422 in the chef studio 44, the multimodal three-dimensionalsensors 20 scan the kitchen module volume to define xyz coordinatesposition and orientation of the standardized kitchen equipment and allobjects therein, whether static or dynamic. At step 1424, the multimodalthree-dimensional sensors 20 scan the kitchen module's volume to findxyz coordinates position of non-standardized objects, such asingredients. At step 1426, the computer 16 creates three-dimensionalmodels for all non-standardized objects and stores their type andattributes (size, dimensions, usage, etc.) in the computer's systemmemory, either on a computing device or on a cloud computingenvironment, and defines the shape, size and type of thenon-standardized objects. At step 1428, the chef movements recordingmodule 98 is configured to sense and capture the chef's arm, wrist andhand movements via the chef's gloves in successive time intervals(chef's hand movements preferably identified and classified according tostandard mini-manipulations). At step 1430, the computer 16 stores thesensed and captured data of the chef's movements in preparing a fooddish into a computer's memory storage device(s).

FIG. 53 is a flow diagram illustrating one embodiment of the process1440 of one embodiment of a robotic apparatus preparing a food dish. Atstep 1442, the multimodal three-dimensional sensors 20 in the robotickitchen 48 scan the kitchen module's volume to find xyz positioncoordinates of non-standardized objects (ingredients, etc.). At step1444, the multimodal three-dimensional sensors 20 in the robotic kitchen48 create three-dimensional models for non-standardized objects detectedin the standardized robotic kitchen 50 and store the shape, size andtype of non-standardized objects in the computer's memory. At step 1446,the robotic cooking module 110 starts a recipe's execution according toa converted recipe file by replicating the chef's food preparationprocess with the same pace, with the same movements, and with similartime duration. At step 1448, the robotic apparatus executes the roboticinstructions of the converted recipe file with a combination of one ormore mini-manipulations and action primitives, thereby resulting in therobotic apparatus in the robotic standardized kitchen preparing the fooddish with the same result or substantially the same result as if thechef 49 had prepared the food dish himself or herself.

FIG. 54 is a flow diagram illustrating the process of one embodiment inthe quality and function adjustment 1450 in obtaining the same orsubstantially the same result in a food dish preparation by a roboticrelative to a chef. At step 1452, the quality check module 56 isconfigured to conduct a quality check by monitoring and validating therecipe replication process by the robotic apparatus via one or moremultimodal sensors, sensors on the robotic apparatus, and usingabstraction software to compare the output data from the roboticapparatus against the controlled data from the software recipe filecreated by monitoring and abstracting the cooking processes carried outby the human chef in the chef studio version of the standardized robotickitchen while executing the same recipe. In step 1454, the robotic foodpreparation engine 56 is configured to detect and determine anydifference(s) that would require the robotic apparatus to make anadjustment to the food preparation process, such as at least monitoringfor the difference in the size, shape, or orientation of an ingredient.If there is a difference, the robotic food preparation engine 56 isconfigured to modify the food preparation process by adjusting one ormore parameters for that particular food dish processing step based onthe raw and processed sensory input data. A determination for acting ona potential difference between the sensed and abstracted processprogress compared to the stored process variables in the recipe scriptis made in step 1454. If the process results of the cooking process inthe standardized robotic kitchen are identical to those spelled out inthe recipe script for the process step, the food preparation processcontinues as described in the recipe script. Should a modification oradaptation to the process be required based on raw and processed sensoryinput data, the adaptation process 1556 is carried out by adjusting anyparameters needed to ensure the process variables are brought intocompliance with those prescribed in the recipe script for that processstep. Upon successful conclusion of the adaptation process 1456, thefood preparation process 1458 resumes as specified in the recipe scriptsequence.

FIG. 55 depicts a flow diagram illustrating a first embodiment in theprocess 1460 of the robotic kitchen preparing a dish by replicating achef's movements from a recorded software file in a robotic kitchen. Instep 1462, a user, through a computer, selects a particular recipe forthe robotic apparatus to prepare the food dish. In step 1464, therobotic food preparation engine 56 is configured to retrieve theabstracted recipe for the selected recipe for food preparation. In step1468, the robotic food preparation engine 56 is configured to upload theselected recipe script into the computer's memory. In step 1470, therobotic food preparation engine 56 calculates the ingredientavailability and the required cooking time. In step 1472, the roboticfood preparation engine 56 is configured to raise an alert ornotification if there is a shortage of ingredients or insufficient timeto prepare the dish according to the selected recipe and servingschedule. The robotic food preparation engine 56 sends an alert to placemissing or insufficient ingredients on a shopping list or selects analternate recipe in step 1472. The recipe selection by the user isconfirmed in step 1474. In step 1476, the robotic food preparationengine 1476 is configured to check whether it is time to start preparingthe recipe. The process 1460 pauses until the start time has arrived instep 1476. In step 1460, the robotic apparatus inspects each ingredientfor freshness and condition (e.g. purchase date, expiration date, odor,color). In step 1462, robotic food preparation engine 56 is configuredto send instructions to the robotic apparatus to move food oringredients from standardized containers to the food preparationposition. In step 1464, the robotic food preparation engine 56 isconfigured to instruct the robotic apparatus to start food preparationat the start time “0” by replicating the food dish from the softwarerecipe script file. In step 1466, the robotic apparatus in thestandardized kitchen 50 replicates the food dish with the same movementas the chef's arms and fingers, the same ingredients, with the samepace, and using the same standardized kitchen equipment and tools. Therobotic apparatus in step 1468 conducts quality checks during the foodpreparation process to make any necessary parameter adjustment. In step1470, the robotic apparatus has completed replication and preparation ofthe food dish, and therefore is ready to plate and serve the food dish.

FIG. 56 depicts the process of storage container check-in andidentification 1480. Using the quality-monitoring dashboard, the userselects to check in an ingredient in step 1482. In step 1484 the userthen scans the ingredient package at the check-in station or counter.Using additional data from the bar code scanner, weighing scales, cameraand laser-scanners, the robotic cooking engine processes theingredient-specific data and maps the same to its ingredient and recipelibrary and analyzes it for any potential allergic impact in step 1486.Should an allergic potential exist based on step 1488, the system instep 1490 decides to notify the user and dispose of the ingredient forsafety reasons. Should the ingredient be deemed acceptable, it is loggedand confirmed by the system in step 1492. The user may in step 1494unpack (if not unpacked already) and drop off the item. In thesucceeding step 1496, the item is packed (foil, vacuum bag, etc.),labeled with a computer-printed label with all necessary ingredient dataprinted thereon, and moved to a storage container and/or storagelocation based on the results of the identification. At step 1498, therobotic cooking engine then updates its internal database and displaysthe available ingredient in its quality-monitoring dashboard.

FIG. 57 depicts an ingredient's check-out from storage and cookingpreparation process 1500. In the first step 1502, the user selects tocheck out an ingredient using the quality-monitoring dashboard. In step1504 the user selects an item to check out based on a single item neededfor one or more recipes. The computerized kitchen then acts in step 1506to move the specific container containing the selected item from itsstorage location to the counter area. In case the user picks up the itemin step 1508, the user processes the item in step 1510 in one or more ofmany possible ways (cooking, disposal, recycling, etc.), with anyremaining item(s) rechecked back into the system in step 1512, whichthen concludes the user's interactions with the system 1514. In the casethat the robotic arms in a standardized robotic kitchen receive theretrieved ingredient item(s), step 1516 is executed in which the armsand hands inspect each ingredient item in the container against theiridentification data (type, etc.) and condition (expiration date, color,odor, etc.). In a quality-check step 1518, the robotic cooking enginemakes a decision on a potential item mismatch or detected qualitycondition. In case the item is not appropriate, step 1520 causes analert to be raised to the cooking engine to follow-up with anappropriate action. Should the ingredient be of acceptable type andquality, the robotic arms move the item(s) to be used in the nextcooking process stage in step 1522.

FIG. 58 depicts the automated pre-cooking preparation process 1524. Instep 1530 the robotic cooking engine calculates the margin and/or wastedingredient materials based on a particular recipe. Subsequently in step1532, the robotic cooking engine searches all possible techniques andmethods for execution of the recipe with each ingredient. In step 1534the robotic cooking engine calculates and optimizes the ingredient usageand methods for time and energy consumption, particularly for dish(es)requiring parallel multi-task processes. The robotic cooking engine thencreates a multi-level cooking plan 1536 for the scheduled dishes andsends the request for cooking execution to the robotic kitchen system.In the next step 1538, the robotic kitchen system moves the ingredients,cooking/baking ware needed for the cooking processes from its automatedshelving system and assembles the tools and equipment and sets up thevarious work stations in step 1540.

FIG. 59 depicts the recipe design and scripting process 1542. As a firststep 1544, the chef selects a particular recipe, for which he thenenters or edits the recipe data in step 1546, including, but not limitedto, the name and other metadata (background, techniques, etc.). In step1548 the chef enters or edits the necessary ingredients based on thedatabase and associated libraries and enters the respective amounts byweight/volume/units required for the recipe. A selection of thenecessary techniques utilized in the preparation of the recipe is madein step 1550 by the chef, based on those available in the database andthe associated libraries. In step 1552 the chef performs a similarselection, but this time he or she is focused on the choice of cookingand preparation methods required to execute the recipe for the dish. Theconcluding step 1554 then allows the system to create a recipe ID whichwill be useful for later database storage and retrieval.

FIG. 60 depicts the process 1556 of how a user might select a recipe.The first step 1558 entails the user purchasing a recipe or subscribingto a recipe-purchase plan from an online marketplace store by way of acomputer or mobile application, thereby enabling a download of a recipescript capable of being replicated. In step 1560 the user searches theonline database and selects a particular recipe from those purchased oravailable as part of a subscription, based on personal preferencesettings and on-site ingredient availability. As a last step 1562, theuser enters the time and date when he/she would like the dish to beready for serving.

FIG. 61A depicts the process 1570 for the recipe search and purchaseand/or subscription process of an online service portal, or so termedrecipe commerce platform. As a first step a new user has to registerwith the system in step 1572 (selecting age, gender, dining preferences,etc., followed by an overall preferred cooking or kitchen style) beforea user can search and browse recipes by downloading them via an app on ahandheld device or using a TV and/or robotic kitchen module. A user maychoose at step 1574 to search using criteria such as style of recipes1576 (including manually cooked recipes) or based on the particularkitchen or equipment style 1578 (wok, steamer, smoker, etc.). The usercan select or set the search to use predefined criteria in step 1580,and using a filtering step 1582 to narrow down the search space andensuing results. In step 1584 the user selects the recipe from theoffered search results, information and recommendation. The user maychoose to then share, collaborate or confer with cooking buddies or thecommunity online about the choice and next steps in step 1586.

FIG. 61B depicts the continuation from FIG. 61A for the recipe searchand purchase/subscription process for a service portal 1590. A user isprompted in step 1592 to select a particular recipe based on either arobotic cooking approach or a parameter-controlled version of therecipe. In the case of a parameter-controlled based recipe, the systemprovides the required equipment details in step 1594 for such items asall the cookware and appliances as well as the robotic arm requirements,and offers select external links at step 1602 to sources for ingredientsand equipment suppliers for detailed ordering instructions. The portalsystem then executes a recipe-type check 1596, where it allows for adirect download and installation 1598 of the recipe program file on theremote device, or requires the user to enter payment information in step1600 based on a one-off payment or payment on a subscription basis,using one of many possible payment forms (PayPal, BitCoin, credit card,etc.).

FIG. 62 depicts the process 1610 used in the creation of a roboticrecipe cooking application (“App”). As a first step 1612, a developeraccount needs to be created on such places as the App Store, Google Playor Windows Mobile or other such marketplaces, including the provision ofbanking and company information. The user is then prompted in step 1614to obtain and download the most updated Application-Program-Interface(API) documentation specific for each app store. A developer then has tofollow the API-requirements spelled out and create a recipe program instep 1618 that meets the API document requirements. In step 1620 thedeveloper needs to provide a name and other metadata for the recipe thatare suitable and prescribed by the various sites (Apple, Google,Samsung, etc.). Step 1622 requires the developer to upload the recipeprogram and metadata files for approval. The respective marketplacesites then review, test and approve the recipe program in step 1624,after which in step 1626 the respective site(s) list and make availablethe recipe program for online searching, browsing and purchase overtheir purchase interface.

FIG. 63 depicts the process 1628 of purchasing a particular recipe orsubscribing to a recipe delivery plan. As a first step 1630 the usersearches for a particular recipe to order. The user may choose to browseby keyword (step 1632) with results able to be narrowed down usingpreference filters (step 1634), browse using other predefined criteria(step 1636) or even browse based on promotional, newly-released orpre-order basis recipes and even live chef cooking events (step 1638).The search results for recipes are displayed to the user in step 1640.The user may then browse these recipe results and preview each recipe inan audio- or short video-clip as part of step 1642. In step 1644 theuser then chooses a device and operating system and receives a specificdownload link for a particular online marketplace application site.Should the user choose at step to connect to a new provider site in task1648, the site will require the new user to complete an authenticationand agreement step 1650, allowing the site to then download and installsite-specific interface software in task 1652, to allow therecipe-delivery process to continue. The provider site will query withthe user whether to create a robotic cooking shopping list in step 1646,and, if agreed to by the user in step 1654, to select a particularrecipe on a single or subscription basis and pick a particular date andtime for the dish to be served. In step 1656 the shopping list for theneeded ingredients and equipment is provided and displayed to the user,including closest and fastest suppliers and their locations, ingredientand equipment availability and associated delivery lead times andpricing. In step 1658 the user is offered a chance to review each of theitems' descriptions and their default or recommended source and brand.The user is then able to view the associated cost of all items on theingredient and equipment list including all associated line-item costs(shipping, tax, etc.) in step 1660. Should the user or buyer want toview alternatives to the proposed shopping list items in step 1662, astep 1664 is executed to offer the user or buyer links to alternatesources to allow them to connect and view alternative buying andordering options. If the user or buyer accepts the proposed shoppinglist, the system not only saves these selections as personalized choicesfor future purchases (step 1666) and updates the current shopping list(step 1668), but then also moves to step 1670, where it selects thealternatives from the shopping list based on additional criteria such aslocal/closest providers, item availability based on season andmaturation-stage, or even pricing for equipment from different supplierswhich has effectively the same performance but differs substantially indelivered cost to the user or buyer.

FIGS. 64A-B are block diagrams illustrating an example of a predefinedrecipe search criterion 1672. The predefined recipe search criteria inthis example include categories like main ingredients, cooking duration,cuisine by geographic regions and types, chef's name search, signaturedishes, and estimated ingredient cost to prepare a food dish. Otherpossible recipe search fields Include types of meals, special diet,exclusion ingredient, dish types and cooking methods, occasions andseasons, reviews and suggestions, and rankings.

FIG. 66 is a block diagram illustrating some pre-defined containers inthe robotic standardized kitchen 50. Each of the containers in thestandardized robotic kitchen 50 has a container number or bar code whichreference the specific content that is stored in that container. Forexample, the first container stores large and bulky products, such aswhite cabbage, red cabbage, savoy cabbage, turnips and cauliflower. Thesixth container stores a large fraction of solids by pieces includingitems like almond shavings, seeds (sunflower, pumpkin, white), driedapricots pitted, dried papaya and dried apricots. FIG. 66 is a blockdiagram illustrating a first embodiment of a robotic restaurant kitchenmodule configured in a rectangular layout with multiple pairs of robotichands for simultaneous food preparation processing. Another embodimentof the invention revolves around a staged configuration for multiplesuccessive or parallel robotic arm and hand stations in a professionalor restaurant kitchen setup shown in FIG. 66. The embodiment depicts amore linear configuration, even though any geometric arrangement couldbe used, showing multiple robotic arm/hand modules, each focused oncreating a particular element, dish or recipe script step (e.g. sixpairs of robotic arms/hands to serve different roles in a commercialkitchen such as sous-chef, broiler-cook, fry/saute cook, pantry cook,pastry chef, soup and sauce cook, etc.). The robotic kitchen layout issuch that the access/interaction with any human or between neighboringarm/hand modules is along a single forward-facing surface. The setup iscapable of being computer-controlled, thereby allowing the entiremulti-arm/hand robotic kitchen setup to perform replication cookingtasks respectively, regardless of whether the arm/hand robotic modulesexecute a single recipe sequentially (end-product from one station getssupplied to the next station for a subsequent step in the recipe script)or multiple recipes/steps in parallel (such as pre-mealfood-/ingredient-preparation for later use during dish replicationcompletion to meet the time crunch during rush times).

FIG. 67 is a block diagram illustrating a second embodiment of a roboticrestaurant kitchen module configured in a U-shape layout with multiplepairs of robotic hands for simultaneous food preparation processing. Yetanother embodiment of the invention revolves around another stagedconfiguration for multiple successive or parallel robotic arm and handstations in a professional or restaurant kitchen setup shown in FIG. 67.The embodiment depicts a rectangular configuration, even though anygeometric arrangement could be used, showing multiple robotic arm/handmodules, each focused on creating a particular element, dish or recipescript step. The robotic kitchen layout is such that theaccess/interaction with any human or between neighboring arm/handmodules is both along a U-shaped outward-facing set of surfaces andalong the central-portion of the U-shape, allowing arm/hand modules topass/reach over to opposing work areas and interact with their opposingarm/hand modules during the recipe replication stages. The setup iscapable of being computer-controlled, thereby allowing the entiremulti-arm/hand robotic kitchen setup to perform replication cookingtasks respectively, regardless of whether the arm/hand robotic modulesexecute a single recipe sequentially (end-product from one station getssupplied to the next station along the U-shaped path for a subsequentstep in the recipe script) or multiple recipes/steps in parallel (suchas pre-meal food-/ingredient-preparation for later use during dishreplication completion to meet the time crunch during rush times, withprepared ingredients possibly stored in containers or appliances(fridge, etc.) contained within the base of the U-shaped kitchen).

FIG. 68 depicts a second embodiment of a robotic food preparation system1680. The chef studio with the standardized robotic kitchen system 1682includes the human chef 49 preparing or executing a recipe, whilesensors on the cookware 1682 record important variables (temperature,etc.) over time and store them in a computer's memory 1684 as sensorcurves and parameters that form a part of a recipe script raw data file.These stored sensory curves and parameter data files from the chefstudio 1682 are delivered to a standardized (remote) robotic kitchen ona purchase or subscription basis 1686. The standardized robotic kitchen1688 installed in a household includes both the user 60 and the computercontrolled system 1690 to operate the automated and/or robotic kitchenequipment based on the received raw data corresponding to the measuredsensory curves and parameter data files.

FIG. 69 depicts another embodiment of the standardized robotic kitchen48. The computer 16 that runs the robotic cooking (software) engine 56,which includes a cooking operations control module 1692 that processesrecorded, analyzed and abstracted sensory data from the recipe script,and associated storage media and memory 1694 to store software filesconsisting of sensory curves and parameter data, interfaces withmultiple external devices. These external devices include, but are notlimited to, a retractable safety glass 68, a computer-monitored andcomputer-controllable storage unit 88, multiple sensors reporting on theprocess of raw-food quality and supply 198, hard-automation modules 82to dispense ingredients, standardized containers 86 with ingredients,and intelligent cookware 1700 fitted with sensors.

FIG. 71 depicts an intelligent cookware item 1700 (a sauce-pot in thisimage) that includes built-in real-time temperature sensors, capable ofgenerating and wirelessly transmitting a temperature profile across thebottom surface of the unit across at least, but not limited to, threeplanar zones, including zone-1 1702, zone-2 1704 and zone-3 1706,arranged in concentric circles across the entire bottom surface of thecookware unit. Each of these three zones is capable of wirelesslytransmitting respective data-1 1708, data-2 1710 and data-3 1712 basedon coupled sensors 1716-1, 1716-2, 1716-3, 1716-4 and 1716-5.

FIG. 71 depicts a typical set of sensory curves 220 with recordedtemperature profiles for data-1 1720, data-2 1722 and data-3 1724, eachcorresponding to the temperature in each of the three zones at thebottom of a particular area of a cookware unit. The measurement unitsfor time are reflected as cooking time in minutes from start to finish(independent variable), while the temperature is measured in degreesCelsius (dependent variable).

FIG. 72 depicts a multiple set of sensory curves 1730 with recordedtemperature 1732 and humidity 1734 profiles, with the data from eachsensor represented as data-1 1736, data-2 1738 all the way to data-N1740. Streams of raw data are forwarded and processed to and by theoperating control unit 274. The measurement units for time are reflectedas cooking time in minutes from start to finish (independent variable),while the temperature and humidity values are measured in degreesCelsius and relative humidity, respectively (dependent variables).

FIG. 73 depicts a process setup for real-time temperature control 1700with a smart (frying) pan. A power source 1750 uses three separatecontrol units, but need not be limited to such, including control-unit-11752, control-unit-2 1754 and control-unit-3 1756, to actively heat aset of inductive coils. The control is in effect a function of themeasured temperature values within each of the (three) zones 1758 (Zone1), 1760 (Zone 2) and 1762 (Zone 3) of the (frying) pan, wheretemperature sensors 1770 (Sensor 1), 1772 (Sensor 2) and 1774 (Sensor 3)wirelessly provide temperature data via data streams 1776 (Data 1), 1778(Data 2) and 1780 (Data 3) back to the operating control unit 274, whichin turn directs the power source 1750 to independently control theseparate zone-heating control units 1752, 1754 and 1756. The goal is toachieve and replicate the desired temperature curves over time as thesensory curve data logged during the human chef's certain (frying) stepduring the preparation of a dish.

FIG. 74 depicts a smart oven and computer control system that arecoupled to the operating control unit 1790, allowing it to execute inreal time a temperature profile for the oven appliance 1792, based on apreviously stored sensory (temperature) curve. The operating controlunit 1790 is able to control the doors (open/close) of the oven, track atemperature profile provided to it by a sensory curve, and,post-cooking, also self-clean. The temperature and humidity inside theoven are monitored through built-in temperature sensors 1794 in variouslocations generating a data stream 268 (Data 1), a temperature sensor inthe form of a probe inserted into the ingredient to be cooked (meat,poultry, etc.) to monitor cooked temperature to infer degree of cookingcompletion, and additional humidity sensors 1796 creating a data stream.The operating control unit 1790 takes in all this sensory data andadjusts the oven parameters to allow it to properly track the sensorycurves described in a previously stored and downloaded set of sensorycurves for both (dependent) variables.

FIG. 75 depicts a computer-controlled ignition and control system setup1798 for a control unit that modulates electric power 1858 to a charcoalgrill to properly trace a sensory curve for one or more temperature andhumidity sensors internally distributed inside the charcoal grill. Thepower control unit 1800 uses electronic control signals 1802 to startthe grill, and signals 1804 and 1806 to adjust the grill-surfacedistance to the charcoal and the injection of water mist 1808 over thecharcoal 1810, to adjust the temperature and humidity of the movable(up/down) rack 1812, respectively. The control unit 1800 bases itsoutput signals 1804,1806 on a set of (five pictured here) data streams1814 for humidity measurement 1816, 1818, 1820, 1822, 1824 from a set ofdistributed humidity sensors (1 through 5) 1826, 1828, 1830, 1832 and1834 inside the charcoal grill, as well as data streams 1836 fortemperature measurements 1840, 1842, 1844, 1846 and 1846 fromdistributed temperature sensors (1 through 5) 1848, 1850, 1852, 1854 and1856.

FIG. 76 depicts a computer-controlled faucet 1860 to allow the computerto control flow rate, temperature and pressure of water fed by thefaucet into the sink (or cookware). The faucet is controlled by acontrol unit 1862 that receives separate data streams 1862 (Data 1),1864 (Data 2) and 1866 (Data 3), which correspond to water flow ratesensor 1868 providing Data 1, temperature sensor 1870 providing Data 2,and water pressure sensor 1872 providing Data 3 sensory data. Thecontrol unit 1862 then controls the supply of cold water 1874, withappropriate cold-water temperature and pressure displayed digitally ondisplay 1876, and hot water 1878, with appropriate hot-water temperatureand pressure displayed digitally on display 1880, to achieve the desiredpressure, flow rate and temperature of water exiting at the spigot.

FIG. 77 depicts an embodiment of a fully instrumented robotic kitchen1882 in top plan view. The standardized robotic kitchen is divided in tothree levels, namely the top level, the counter level and the lowerlevel, with each level containing equipment and appliances that haveintegrally mounted sensors 1884 and computer-control units 1886.

The top level contains multiple cabinet-type modules with differentunits to perform specific kitchen functions by way of built-inappliances and equipment. At the simplest level a shelf/cabinet storagearea 82 is included, a cabinet volume 1320 used for storing andaccessing cooking tools and utensils and other cooking and serving ware(cooking, baking, plating, etc.), a storage ripening cabinet volume 1298for particular ingredients (e.g. fruit and vegetables, etc.), a chilledstorage zone 88 for such items as lettuce and onions, a frozen storagecabinet volume 1302 for deep-frozen items, and another storage pantryzone 1304 for other ingredients and rarely used spices, etc. Each of themodules within the top level contains sensor units 1884 providing datato one or more control units 1886, either directly or by way of one ormore central or distributed control computers, to allow forcomputer-controlled operations.

The counter level not only houses monitoring sensors 1884 and controlunits 1886, but also includes a serving counter 1306, a counter areawith a sink 1308, another counter area 1310 with removable workingsurfaces (cutting/chopping board, etc.), a charcoal-based slatted grill1312 and a multi-purpose area for other cooking appliances 1314,including a stove, cooker, steamer and poacher. Each of the moduleswithin the counter level contains sensor units 1884 providing data toone or more control units 1886, either directly or by way of one or morecentral or distributed control computers, to allow forcomputer-controlled operations.

The lower level houses the combination convection oven and microwave aswell as steamer, poacher and grill 1316, the dish-washer 1318, the hardautomation controlled ingredient dispensers 82, and a larger cabinetvolume 1320 that holds and stores additional frequently used cooking andbaking ware, as well as tableware, flatware, utensils (whisks, knives,etc.) and cutlery. Each of the modules within the lower level containssensor units 1884 providing data to one or more control units 1886,either directly or by way of one or more central or distributed controlcomputers, to allow for computer-controlled operations.

FIG. 78 depicts a perspective view of one embodiment of a robotickitchen cooking system 1890, with three different levels arranged fromtop to bottom, each fitted with multiple and distributed sensor units1892 which feed data directly to one or more control units 1894, or toone or more central computers, which in turn use and process the sensorydata to then command one or more control units 376 to act on theircommands.

The top level contains multiple cabinet-type modules with differentunits to perform specific kitchen functions by way of built-inappliances and equipment. At the simplest level a shelf/cabinet storagepantry volume 1294 is included, a cabinet volume 1296 used for storingand accessing cooking tools and utensils and other cooking and servingware (cooking, baking, plating, etc.), a storage ripening cabinet volume1298 for particular ingredients (e.g. fruit and vegetables, etc.), achilled storage zone 88 for such items as lettuce and onions, a frozenstorage cabinet volume 1302 for deep-frozen items, and another storagepantry zone 1294 for other ingredients and rarely used spices, etc. Eachof the modules within the top level contains sensor units 1892 providingdata to one or more control units 1894, either directly or by way of oneor more central or distributed control computers, to allow forcomputer-controlled operations.

The counter level not only houses monitoring sensors 1892 and controlunits 1894, but also includes a counter area with a sink andelectronically controllable faucet 1308, another counter area 1310 withremovable working surfaces for cutting/chopping on a board, etc., acharcoal-based slatted grill 1312, and a multi-purpose area for othercooking appliances 1314, including a stove, cooker, steamer and poacher.Each of the modules within the counter level contains sensor units 1892providing data to one or more control units 1894, either directly or byway of one or more central or distributed control computers, to allowfor computer-controlled operations.

The lower level houses the combination convection oven and microwave aswell as steamer, poacher and grill 1316, the dish-washer 1318, the hardautomation controlled ingredient dispensers 82, and a larger cabinetvolume 1310 that holds and stores additional frequently used cooking andbaking ware, as well as tableware, flatware, utensils (whisks, knives,etc.) and cutlery. Each of the modules within the lower level containssensor units 1892 providing data to one or more control units 1896,either directly or by way of one or more central or distributed controlcomputers, to allow for computer-controlled operations.

FIG. 79 is a flow diagram illustrating a second embodiment 1900 in theprocess of the robotic kitchen preparing a dish from one or morepreviously recorded parameter curves in a standardized robotic kitchen.In step 1902, a user, through a computer, selects a particular recipefor the robotic apparatus to prepare the food dish. In step 1904, therobotic food preparation engine is configured to retrieve the abstractedrecipe for the selected recipe for food preparation. In step 1906, therobotic food preparation engine is configured to upload the selectedrecipe script into the computer's memory. In step 1908, the robotic foodpreparation engine calculates the ingredient availability. In step 1910,the robotic food preparation engine is configured to evaluate whetherthere is a shortage or a absence of ingredients to prepare the dishaccording to the selected recipe and serving schedule. The robotic foodpreparation engine sends an alert to place missing or insufficientingredients on a shopping list or selects an alternate recipe in step1912. The recipe selection by the user is confirmed in step 1914. Instep 1916, the robotic food preparation engine is configured to sendrobotic instructions to the user to place food or ingredients intostandardized containers and move them to the proper food preparationposition. In step 1918, the user is given the option to select areal-time video-monitor projection, whether on a dedicated monitor or aholographic laser-based projection, to visually see each and every stepof the recipe replication process based on all movements and processesexecuted by the chef while being recorded for playback in this instance.In step 1920, the robotic food preparation engine is configured to allowthe user to start food preparation at start time “0” of their choosingand powering on the computerized control system for the standardizedrobotic kitchen. In step 1922 the user executes a replication of all thechef's actions based on the playback of the entire recipe creationprocess by the human chef on the monitor/projection screen, wherebysemi-finished products are moved to designated cookware and appliancesor intermediate storage containers for later use. In step 1924, therobotic apparatus in the standardized kitchen executes the individualprocessing steps according to sensory data curves or based on cookingparameters recorded when the chef executed the same step in the recipepreparation process in the chef studio's standardized robotic kitchen.In step 1926 the robotic food preparation's computer controls all thecookware and appliance settings in terms of temperature, pressure andhumidity so as to replicate the required data curves over the entirecooking time based on the data captured and saved while the chef waspreparing the recipe in the chef's studio standardized robotic kitchen.In step 1928 the user makes all simple movements so as to replicate thechef's steps and process movements as evidenced through the audio andvideo instructions relayed to the user over the monitor or projectionscreen. In step 1930 the robotic kitchen's cooking engine alerts theuser when a particular cooking step based on a sensory curve orparameter set has been completed. Once the user and computer controllerinteractions result in the completion of all cooking steps in therecipe, the robotic cooking engine sends a request to terminate thecomputer-controlled portion of the replication process in step 1932. Instep 1934, the user removes the completed recipe dish, plates and servesit, or continues any remaining cooking steps or processes manually.

FIG. 80 depicts the sensory data capturing process 1936 in the chefstudio. The first step 1938 is for the chef to create or design therecipe. A next step 1940 requires that the chef input the name,ingredients, measurement and process descriptions for the recipe intothe robotic cooking engine. The chef begins by loading all the requiredingredients into designated standardized storage containers, appliancesand select appropriate cookware in step 1942. The next step 1944involves the chef setting the start time and switching on the sensoryand processing systems to record all sensed raw data and allow forprocessing of the same. Once the chef starts cooking in step 1946, allembedded and monitoring sensor units and appliances report and send rawdata to the central computer system to allow it to record in real timeall relevant data during the entire cooking process 1948. Additionalcooking parameters and audible chef comments are further recorded andstored as raw data in step 1950. A robotic cooking module abstraction(software) engine processes all raw data, including two- andthree-dimensional geometric motion and object recognition data, togenerate a machine-readable and machine-executable recipe script as partof step 1952. Upon completion of the chef studio recipe creation andcooking process by the chef, the robotic cooking engine generates asimulation visualization program 1954 replicating the movement and mediadata used for later recipe replication by a remote standardized robotickitchen system. Based on the raw and processed data, and a confirmationof the simulated recipe execution visualization by the chef,hardware-specific applications are developed and integrated fordifferent (mobile) operating systems and submitted to onlinesoftware-application stores and/or marketplaces in step 1956, for directsingle-recipe user purchase or multi-recipe purchase via subscriptionmodels.

FIG. 81 depicts the process and flow of a household robotic cookingprocess 1960. The first step 1962 involves the user selecting a recipeand acquiring the digital form of the recipe. In step 1964 the roboticcooking engine receives the recipe script containing machine-readablecommands to cook the selected recipe. The recipe is uploaded in step1966 to the robotic cooking engine with the script being placed inmemory. Once stored, step 1968 calculates the necessary ingredients anddetermines their availability. In a logic check 1970 the systemdetermines whether to alert the user or send a suggestion in step 1972urging adding missing items to the shopping list or suggesting analternative recipe to suit the available ingredients, or to proceedshould sufficient ingredients be available. Once ingredient availabilityis verified in step 1974, the system confirms the recipe and the user isqueried in step 1976 to place the required ingredients into designatedstandardized containers in a position where the chef started the recipecreation process originally (in the chef studio). The user is promptedto set the start time of the cooking process and to set the cookingsystem to proceed in step 1978. Upon start-up the robotic cooking systembegins the execution of the cooking process 1980 in real time accordingto sensory curves and cooking parameter data provided in the recipescript data files. During the cooking process 1982, the computer, so asto replicate the sensory curves and parameter data files originallycaptured and saved during the chef studio recipe creation process,controls all appliances and equipment. Upon completion of the cookingprocess, the robotic cooking engine sends a reminder based on havingdecided the cooking process is finished in step 1984. Subsequently therobotic cooking engine sends a termination request 1986 to thecomputer-control system to terminate the entire cooking process, and instep 1988 the user removes the dish from the counter for serving orcontinues any remaining cooking steps manually.

FIG. 82 depicts another embodiment of a standardized robotic foodpreparation kitchen system 48. The computer 16 that runs the roboticcooking (software) engine 56, which includes a cooking operationscontrol module 1990 that processes recorded, analyzed and abstractedsensory data from the recipe script, a visual command monitoring module1992, and associated storage media and memory 1994 to store softwarefiles consisting of sensory curves and parameter data, interfaces withmultiple external devices. These external devices include, but are notlimited to, an instrumented kitchen working counter 90, the retractablesafety glass 68, the instrumented faucet 92, cooking appliances withembedded sensors 74, cookware 1700 with embedded sensors (stored on ashelf or in a cabinet), standardized containers and ingredient storageunits 78, a computer-monitored and computer-controllable storage unit88, multiple sensors reporting on the process of raw food quality andsupply 1996, hard automation modules 82 to dispense ingredients, and anoperations control unit 1998.

FIG. 83 depicts an embodiment of a fully instrumented robotic kitchen2000 in top plan view. The standardized robotic kitchen is divided intothree levels, namely the top level, the counter level and the lowerlevel, with each level containing equipment and appliances that haveintegrally mounted sensors 1884 and computer-control units 1886.

The top level contains multiple cabinet-type modules with differentunits to perform specific kitchen functions by way of built-inappliances and equipment. At the simplest level this includes a cabinetvolume 1296 used for storing and accessing cooking tools and utensilsand other cooking and serving ware (cooking, baking, plating, etc.), astorage ripening cabinet volume 1298 for particular ingredients (e.g.fruit and vegetables, etc.), a chilled storage zone 1300 for such itemsas lettuce and onions, a frozen storage cabinet volume 1302 fordeep-frozen items, and another storage pantry zone 1304 for otheringredients and rarely used spices, etc. Each of the modules within thetop level contains sensor units 1884 providing data to one or morecontrol units 1886, either directly or by way of one or more central ordistributed control computers, to allow for computer-controlledoperations.

The counter level not only houses monitoring sensors 1884 and controlunits 1886, but also includes the one or more robotic arms, wrists andmulti-fingered hands 72, a serving counter 1306, a counter area with asink 1308, another counter area 1310 with removable working surfaces(cutting/chopping board, etc.), a charcoal-based slatted grill 1312 anda multi-purpose area for other cooking appliances 1314, including astove, cooker, steamer and poacher. Each of the modules within thecounter level contains sensor units 1884 providing data to one or morecontrol units 1886, either directly or by way of one or more central ordistributed control computers, to allow for computer-controlledoperations.

The lower level houses the combination convection oven and microwave aswell as steamer, poacher and grill 1316, the dish-washer 1318, the hardautomation controlled ingredient dispensers 82, and a larger cabinetvolume 3=378 that holds and stores additional frequently used cookingand baking ware, as well as tableware, flatware, utensils (whisks,knives, etc.) and cutlery. Each of the modules within the lower levelcontains sensor units 1884 providing data to one or more control units1886, either directly or by way of one or more central or distributedcontrol computers, to allow for computer-controlled operations.

FIG. 84 depicts an embodiment of a fully instrumented robotic kitchen2000 in perspective view, with an overlaid coordinate frame designatingthe x-axis 1322, the y-axis 1324 and the z-axis 1326, within which allmovements and locations will be defined and referenced to the origin(0,0,0). The standardized robotic kitchen is divided in to three levels,namely the top level, the counter level and the lower level, with eachlevel containing equipment and appliances that have integrally mountedsensors 1884 and computer-control units 1886.

The top level contains multiple cabinet-type modules with differentunits to perform specific kitchen functions by way of built-inappliances and equipment.

At the simplest level this includes a cabinet volume 1294 used forstoring and accessing standardized cooking tools and utensils and othercooking and serving ware (cooking, baking, plating, etc.), a storageripening cabinet volume 1298 for particular ingredients (e.g. fruit andvegetables, etc.), a chilled storage zone 1300 for such items as lettuceand onions, a frozen storage cabinet volume 86 for deep-frozen items,and another storage pantry zone 1294 for other ingredients and rarelyused spices, etc. Each of the modules within the top level containssensor units 1884 providing data to one or more control units 1886,either directly or by way of one or more central or distributed controlcomputers, to allow for computer-controlled operations.

The counter level not only houses monitoring sensors 1884 and controlunits 1886, but also includes the one or more robotic arms, wrists andmulti-fingered hands 72, a counter area with a sink and electronicfaucet 1308, another counter area 1310 with removable working surfaces(cutting/chopping board, etc.), a charcoal-based slatted grill 1312 anda multi-purpose area for other cooking appliances 1314, including astove, cooker, steamer and poacher. Each of the modules within thecounter level contains sensor units 1884 providing data to one or morecontrol units 1886, either directly or by way of one or more central ordistributed control computers, to allow for computer-controlledoperations.

The lower level houses the combination convection oven and microwave aswell as steamer, poacher and grill 1315, the dish-washer 1318, the hardautomation controlled ingredient dispensers 82 (not shown), and a largercabinet volume 1310 that holds and stores additional frequently usedcooking and baking ware, as well as tableware, flatware, utensils(whisks, knives, etc.) and cutlery. Each of the modules within the lowerlevel contains sensor units 1884 providing data to one or more controlunits 1886, either directly or by way of one or more central ordistributed control computers, to allow for computer-controlledoperations.

FIG. 85 depicts an embodiment of a fully instrumented robotic kitchen2020 in top plan view. The standardized robotic kitchen is divided intothree levels, namely the top level, the counter level and the lowerlevel, with the top and lower levels containing equipment and appliancesthat have integrally mounted sensors 1884 and computer-control units1886, and the counter level being fitted with one or more command andvisual monitoring devices 2022.

The top level contains multiple cabinet-type modules with differentunits to perform specific kitchen functions by way of built-inappliances and equipment. At the simplest level this includes a cabinetvolume 1296 used for storing and accessing standardized cooking toolsand utensils and other cooking and serving ware (cooking, baking,plating, etc.), a storage ripening cabinet volume 1298 for particularingredients (e.g. fruit and vegetables, etc.), a chilled storage zone1300 for such items as lettuce and onions, a frozen storage cabinetvolume 1302 for deep-frozen items, and another storage pantry zone 1304for other ingredients and rarely used spices, etc. Each of the moduleswithin the top level contains sensor units 1884 providing data to one ormore control units 1886, either directly or by way of one or morecentral or distributed control computers, to allow forcomputer-controlled operations.

The counter level houses not only monitoring sensors 1884 and controlunits 1886, but also visual command monitoring devices 2020 while alsoincluding a serving counter 1306, a counter area with a sink 1308,another counter area 1310 with removable working surfaces(cutting/chopping board, etc.), a charcoal-based slatted grill 1312 anda multi-purpose area for other cooking appliances 1314, including astove, cooker, steamer and poacher. Each of the modules within thecounter level contains sensor units 1884 providing data to one or morecontrol units 1886, either directly or by way of one or more central ordistributed control computers, to allow for computer-controlledoperations. Additionally, one or more visual command monitoring devices2022 are also provided within the counter level for the purposes ofmonitoring the visual operations of the human chef in the studio kitchenas well as the robotic arms or human user in the standardized robotickitchen, where data is fed to one or more central or distributedcomputers for processing and subsequent corrective or supportivefeedback and commands sent back to the robotic kitchen for display orscript-following execution.

The lower level houses the combination convection oven and microwave aswell as steamer, poacher and grill 1316, the dish-washer 1318, the hardautomation controlled ingredient dispensers 86 (not shown), and a largercabinet volume 1320 that holds and stores additional frequently usedcooking and baking ware, as well as tableware, flatware, utensils(whisks, knives, etc.) and cutlery. Each of the modules within the lowerlevel contains sensor units 1884 providing data to one or more controlunits 1886, either directly or by way of one or more central ordistributed control computers, to allow for computer-controlledoperations.

FIG. 86 depicts an embodiment of a fully instrumented robotic kitchen2020 in perspective view. The standardized robotic kitchen is dividedinto three levels, namely the top level, the counter level and the lowerlevel, with the top and lower levels containing equipment and appliancesthat have integrally mounted sensors 1884 and computer-control units1886, and the counter level being fitted with one or more command andvisual monitoring devices 2022.

The top level contains multiple cabinet-type modules with differentunits to perform specific kitchen functions by way of built-inappliances and equipment. At the simplest level this includes a cabinetvolume 1296 used for storing and accessing standardized cooking toolsand utensils and other cooking and serving ware (cooking, baking,plating, etc.), a storage ripening cabinet volume 1298 for particularingredients (e.g. fruit and vegetables, etc.), a chilled storage zone1300 for such items as lettuce and onions, a frozen storage cabinetvolume 86 for deep-frozen items, and another storage pantry zone 1294for other ingredients and rarely used spices, etc. Each of the moduleswithin the top level contains sensor units 1884 providing data to one ormore control units 1886, either directly or by way of one or morecentral or distributed control computers, to allow forcomputer-controlled operations.

The counter level houses not only monitoring sensors 1884 and controlunits 1886, but also visual command monitoring devices 1316 while alsoincluding a counter area with a sink and electronic faucet 1308, anothercounter area 1310 with removable working surfaces (cutting/choppingboard, etc.), a (smart) charcoal-based slatted grill 1312 and amulti-purpose area for other cooking appliances 1314, including a stove,cooker, steamer and poacher. Each of the modules within the counterlevel contains sensor units 1184 providing data to one or more controlunits 1186, either directly or by way of one or more central ordistributed control computers, to allow for computer-controlledoperations. Additionally, one or more visual command monitoring devices(not shown) are also provided within the counter level for the purposesof monitoring the visual operations of the human chef in the studiokitchen as well as the robotic arms or human user in the standardizedrobotic kitchen, where data is fed to one or more central or distributedcomputers for processing and subsequent corrective or supportivefeedback and commands sent back to the robotic kitchen for display orscript-following execution.

The lower level houses the combination convection oven and microwave aswell as steamer, poacher and grill 1316, the dish-washer 1318, the hardautomation controlled ingredient dispensers 86 (not showed)s, and alarger cabinet volume 1309 that holds and stores additional frequentlyused cooking and baking ware, as well as tableware, flatware, utensils(whisks, knives, etc.) and cutlery. Each of the modules within the lowerlevel contains sensor units 1307 providing data to one or more controlunits 376, either directly or by way of one or more central ordistributed control computers, to allow for computer-controlledoperations.

FIG. 87A depicts another embodiment of a standardized robotic kitchensystem 48. The computer 16 that runs the robotic cooking (software)engine 56 and a memory module 102 for storing recipe script data andsensory curves and parameter data files, interfaces with multipleexternal devices. These external devices include, but are not limitedto, instrumented robotic kitchen stations 2030, instrumented servingstations 2032, an instrumented washing and cleaning station 2034,instrumented cookware 2036, computer-monitored and computer-controllablecooking appliances 2038, special-purpose tools and utensils 2040, anautomated shelf station 2042, an instrumented storage station 2044, aningredient retrieval station 2046, a user console interface 2048, dualrobotic arms 70, hard automation modules 82 to dispense ingredients, anda chef-recording device 2050.

FIG. 87B depicts one embodiment of a robotic kitchen cooking system 2060in plan view, where the chef 49 or home-cook user 60 can access variouscooking stations from multiple (four shown here) sides. A centralstorage station 2062 provides for different storage areas for variousfood items held at different temperatures (chilled/frozen) for optimumfreshness, allowing access from all sides. Along the perimeter of thesquare arrangement of the current embodiment, the chef 49 or user 60 canaccess various cooking areas with modules that include, but are notlimited to, a user/chef console 2064 for laying out the recipe andoverseeing the processes, an ingredient access station 2066 including ascanner, camera and other ingredient characterization systems, anautomatic shelf station 2068 for cookware/baking ware/tableware, awashing and cleaning station 2070 consisting of at least a sink anddish-washer unit, a specialized tool and utensil station 2072 forspecialized tools required for particular techniques used in food oringredient preparation, a warming station 2074 for warming or chillingserved dishes and a cooking appliance station 2076 consisting ofmultiple appliances including, but not limited to, an oven, stove,grill, steamer, fryer, microwave, blender, dehydrator, etc.

FIG. 87C depicts a perspective view of the same embodiment of a robotickitchen 48, allowing a chef 49 or a user 60 to gain access to multiplecooking stations and equipment from at least four different sides. Acentral storage station 2062 provides for different storage areas forvarious food items held at different temperatures (chilled/frozen) foroptimum freshness, allowing access from all sides, and is located at anelevated level. An automatic shelf station 2068 for cookware/bakingware/tableware is located at a middle level beneath the central storagestation 2062. At a lower level an arrangement of cooking stations andequipment is located that includes, but is not limited to, a user/chefconsole 2064 for laying out the recipe and overseeing the processes, aningredient access station 2060 including a scanner, camera and otheringredient characterization systems, an automatic shelf station 2068 forcookware/baking ware/tableware, a washing and cleaning station 2070consisting of at least a sink and dish-washer unit, a specialized tooland utensil station 2072 for specialized tools required for particulartechniques used in food or ingredient preparation, a warming station2076 for warming or chilling served dishes and a cooking appliancestation 2076 consisting of multiple appliances including, but notlimited to, an oven, stove, grill, steamer, fryer, microwave, blender,dehydrator, etc.

FIG. 88 is a block diagram Illustrating a robotic human-emulatorelectronic intellectual property (IP) library 2100. The robotichuman-emulator electronic IP library 2100 covers the various concepts inwhich the robotic apparatus is used as a means to replicate a human'sparticular skill set. More specifically, the robotic apparatus, whichincludes the pair of robotic hands 70 and the robotic arms 72, serves toreplicate a set of specific human skills. In some way, the transfer tointelligence from a human can be captured through the use of the human'shands; the robotic apparatus then replicates the precise movements ofthe recorded movements in obtaining the same result. The robotichuman-emulator electronic IP library 2100 includes a robotichuman-culinary-skill replication engine 56, a robotichuman-painting-skill replication engine 2102, a robotichuman-musical-instrument-skill replication engine 2102, a robotichuman-nursing-care-skill replication engine 2104, a robotichuman-emotion recognizing engine 2106, a robotic human-intelligencereplication engine 2108, an input/output module 2110, and acommunication module 2112. The robotic human emotion recognizing engine1358 is further described with respect to FIGS. 90, 91, 92 and 92.

FIG. 89 is a robotic human-emotion recognizing (or response) engine2106, which includes a training block coupled to an application blockvia the bus 2120. The training block contains a human input stimulimodule 2122, a sensor module 2124, a human emotion response module (toinput stimuli) 2126, an emotion response recording module 2128, aquality check module 2130, and a learning machine module 2132. Theapplication block contains an input analysis module 2134, a sensormodule 2136, a response generating module 2138, and a feedbackadjustment module 2140.

FIG. 90 is a flow diagram illustrating the process and logic flow of arobotic human emotion system 2150. In its first step 2151, the(software) engine receives sensory input from a variety of sources akinto the senses of a human, including vision, audible feedback, tactileand olfactory sensor data from the surrounding environment. In thedecision step 2152, the decision is made whether to create a motionreflex, either resulting in a reflex motion 2153 or, if no reflex motionis required, step 2154 is executed, where specific input information orpatterns or combinations thereof are recognized based on information orpatterns stored in memory, which are subsequently translated intoabstract or symbolic representations. The abstract and/or symbolicinformation is processed through a sequence of intelligence loops, whichcan be experience-based. Another decision step 2156 decides on whether amotion-reaction 2157 should be engaged based on a known and pre-definedbehavior model and, if not, step 12158 is undertaken. In step 2158 theabstract and/or symbolic information is then processed through anotherlayer of emotion- and mood-reaction behavior loops with inputs providedfrom internal memories, which can be formed through learning. Emotion isbroken down into a mathematical formalism and programmed into robot,with mechanisms that can be described, and quantities that can bemeasured and analyzed (e.g. by capturing facial expressions of howquickly a smile forms and how long it lasts to differentiate between agenuine and a polite smile, or by detecting emotion based on the vocalqualities of a speaker, where the computer measures the pitch, energyand volume of the voice, as well as the fluctuations in volume and pitchfrom one moment to the next). There will thus be certain identifiableand measurable metrics to an emotional expression, where these metricsin the behavior of an animal or the sound of a human speaking or singingwill have identifiable and measurable associated emotion attributes.Based on these identifiable and measurable metrics, the emotion enginecan make a decision 2159 as to which behavior to engage, whetherpre-learned or newly learned. The engaged or executed behavior and itseffective result are updated in memory and added to the experiencepersonality and natural behavior database 2160. In a follow-on step2161, the experience personality data is translated into morehuman-specific information, which then allows him or her to execute theprescribed or resultant motion 2162.

FIGS. 91A-C are flow diagrams illustrating the process of comparing aperson's emotional profile against a population of emotional profileswith hormones, pheromones and other. FIG. 91A describes the process ofthe emotional profile application, where a person's emotion parametersare monitored and extracted in 2182 from a user's general profile 2184,and based on a stimulus input, parameter value changes from a baselinevalue derived from a segmented timeline, taken and compared to those foran existing larger group under similar conditions. Ate step 1804, Firstlevel degrouping based on one or more criteria parameters (e.g., degroupbased on the speed of change of people with the same emotionalparameters). The process continues the emotion parameter degrouping andsegregation into further steps of emotional parameter comparisons, whichcan include continued levels represented by a set of pheromones 1808, aset of micro-expressions 1809, the person's heart rate and perspiration1810, pupil dilation 1811, observed reflexive movements 1812, awarenessof overall body temperature 1813, and perceived situational pressure1814. The degrouped emotion parameters are then used to determine asimilar grouping of parameters 1815 for comparison purposes.

FIG. 91B depicts all the individual emotion groupings such as immediateemotions 1820 such as anger, secondary emotions 1821 such as fear, allthe way through to N actual emotions. The next step 1823 then computesthe associated emotion(s) in each group according to the associatedemotional profile data, leading to the assessment 1824 of the intensitylevel of the emotional state, which allows the engine to then decide onthe appropriate action 1825.

FIG. 91C depicts the automated process 1830 of mass group emotionalprofile development and learning. The process involves receiving newmulti-source emotional profile and condition inputs from various sources1831, with an associated quality-check of profile/parameter data change1832. The plurality of the emotional profile data is stored in step 1833and, using multiple machine learning techniques 1835, an iterative loop1834 of analyzing and classifying each profile and data set into variousgroupings with matching (sub-)sets in the central database is carriedout.

FIG. 92A is a block diagram illustrating the emotional detection andanalysis 2220 of a person's emotional state by monitoring a set ofhormones, a set of pheromones, and other key parameters. A person'semotional state can be detected by monitoring and analyzing the person'sphysiological signs, under a defined condition with internal and/orexternal stimulus, and assessing how these physiological signs changeover a certain timeline. One embodiment of the degrouping process isbased on one or more criteria parameters (e.g., degroup based on thespeed of change of people with the same emotional parameters).

In one embodiment the emotional profile can be detected via machinelearning methods based on statistical classifiers where the inputs areany measured levels of pheromones, hormones, or other features such asvisual or auditory cues. If the set of features is {x₁, x₂, x₃, . . . ,x_(n)} represented as a vector and y represents the emotional state,then the general form of an emotion-detection statistical classifier is:

$y = {\underset{j,l}{argmin}\lbrack {( {\sum\limits_{i}{{y_{i} - {f_{j,p_{l}}( {\overset{->}{x}}_{i} )}}}} ) + {\beta( {f_{j},p_{l}} )}} \rbrack}$Where the function f is a decision tree, a neural network, a logisticregressor, or other statistical classifier described in the machinelearning literature. The first term minimizes the empirical error (theerror detected while training the classifier) and the second termminimizes the complexity—e.g. Occam's razor, finding the simplestfunction and set of parameters p for that function that yield thedesired result.

Additionally, in order to determine which pheromones or other featuresmake the most difference (add the most value) to predicting emotionalstate, an active-learning criterion can be added, generally expressedas:

$\underset{{\overset{->}{x}}_{i} \in {\{{{\overset{->}{x}}_{k + 1},\ldots\mspace{14mu},{\overset{->}{x}}_{n}}\}}}{argmin}( {L( {\hat{f}( {{\overset{->}{x}}_{test},{\hat{y}}_{test}} )} )} \middle| {{\overset{->}{x}}_{i}\bigcup\{ {{\overset{->}{x}}_{1},\ldots\mspace{14mu},{\overset{->}{x}}_{k}} \}} )$Where L is a “loss function”, f is the same statistical classifier as inthe previous equation, and y-hat is the known outcome. We measurewhether the statistical classifier performs better (smaller lossfunction) by addition new features, and if so keep them, otherwise not.

Parameters, values and quantities that evolve over time can be assessedto create a human emotional profile by detecting the change ortransformation from one moment to the next. There are identifiablequalities to an emotional expression. A robot with emotions in responseto its environment could make quicker and more effective decisions, e.g.when a robot is motivated by fear or joy or desire it might make betterdecisions and attain the goals more effectively and efficiently.

The robotic emotion engine replicates the human hormone emotions andpheromone emotions, either individually or in combination. Hormoneemotions refer to how hormones change inside of a person's body and howthat affects a person's emotions. Pheromone emotions refer to pheromonesthat are outside a person's body, such as smell, that affect a person'semotions. A person's emotional profile can be constructed byunderstanding and analyzing the hormone and pheromone emotions. Therobotic emotion engine attempts to understand a person's emotions suchas anger and fear by using sensors to detect a person's hormone andpheromone profile.

There are nine key physiological sign parameters to be measured in orderto build a person's emotional profile: (1) sets of hormones 2221, whichare secreted internally and trigger various biochemical pathways thatcause certain effects, e.g. adrenalin and insulin are hormones, (2) setsof pheromones 2222, which are secreted externally, and have an effect onanother person in a similar way, e.g. androstenol, androstenone andandrostadienone, (3) micro expression 2223, which is a brief,involuntary facial expression shown by humans according to emotionsexperienced, (4) the heart rate 2224 or heart beat, e.g., when aperson's heart rate increases, (5) sweat 2225 (e.g., goose bumps) e.g.face blushes and palms get sweaty and in the state of being excited ornervous, (6) pupil dilation 2226 (and iris sphincter, biliary muscle),e.g. pupil dilation for a short time in response to feelings of fear,(7) reflex movement v7, which is the movement/action primarilycontrolled by the spinal arc, as a response to an external stimulus,e.g. jaw jerk reflex, (8) body temperature 2228 (9) pressure 2229. Theanalysis 2230 on how these parameters change over a certain time 2231may reveal a person's emotional state and profile.

FIG. 92B is a block diagram illustrating a robot 1590 assessing andlearning about a person's emotional behavior. The parameter readings areanalyzed 2240 and divided into emotion and/or non-emotional responses,with internal stimulus 2242 and/or external stimulus 2244, e.g.pupillary light reflex is only at the level of the spinal cord, pupilsize can change when a person is angry, in pain, or in love, whereasinvoluntary responses generally involve the brain as well. Use ofcentral nervous system stimulant drugs and some hallucinogenic drugs cancause dilation of the pupils.

FIG. 93 is a block diagram illustrating a port device 2230 implanted ina person to detect and record the person's emotional profile. Whenmeasuring the physiological signs change, a person can monitor andrecord the emotional profile for a time period by pressing a button witha first tag on the time at which the change of emotion has started andtouch the button again with a second tag when the emotion change hasconcluded. This process enables a computer to assess and learn about aperson's emotional profile based on the change in emotion parameters.With data/information collected from mass amount of users the computerclassifies all changes associated with each emotion and mathematicallyfinds the significant and specific parameter changes that areattributable to particular emotion characteristics.

When a user experiences an emotion or mood swing, physiologicalparameters such as hormone, heart rate, sweat, pheromones can bedetected and recorded with a port connecting to a person's body, abovethe skin and directly to the vein. The start time and end time of themood change can be determined by the person himself or herself as theperson's emotional state changes. For example, a person initiates fourmanual emotion cycles and creates four timelines within a week, and asdetermined by the person, the first one lasts 2.8 hour from the time hetags the start till the time he tags the end. The second cycle last for2 hours, the third one last for 0.8 hours, and the fourth one last for1.6 hours.

FIG. 94A depicts a robotic human-intelligence engine 2250. In thereplication engine 1360, there are two main blocks, including a trainingblock and an application block, both containing multiple additionalmodules all interconnected to each other over a common inter-modulecommunication bus 72. The training block of the human-intelligenceengine contains further modules, including, but not limited to, a sensorinput module 1404, a human input stimuli module 1402, a humanintelligence response module 1420 that reacts to input stimuli, anintelligence response recording module 1422, a quality check module 1410and a learning machine module 1412. The application block of thehuman-intelligence engine contains further modules, including, but notlimited to, an input analysis module 1414, a sensor input module 1404, aresponse generating module 1416, and a feedback adjustment module 1418.

FIG. 94B depicts the architecture of the robotic human intelligencesystem 1136. The system is split into both the cognitive robotic agentand the human skill execution module. Both modules share sensingfeedback data 1482, as well as sensed motion data 1538 and modeledmotion data 1539. The cognitive robotic agent module includes, but isnot necessarily limited to, modules that represent a knowledge database1531, interconnected to an adjustment and revision module 1534, withboth being updated through a learning module 1535. Existing knowledge1532 is fed into the execution monitoring module 1536 as well asexisting knowledge 1533 being fed into the automated analysis andreasoning module 1537, where both receive sensing feedback data 1482from the human skill execution module, with both also providinginformation to the learning module 1535. The human skill executionmodule consists of both a control module 1138 that bases its controlsignals on collecting and processing multiple sources of feedback(visual and auditory), as well as a module 1541 with a robot utilizingstandardized equipment, tools and accessories.

FIG. 95A depicts the architecture for a robotic painting system 1440.Included in this system are both a studio robotic painting system 1441and a commercial robotic painting system 1445, communicatively connected1444 to allow software program files or applications for roboticpainting to be delivered from the studio robotic painting system 1441 tothe commercial robotic painting system 1445 based on a single-unitpurchase or subscription-based payment basis. The studio roboticpainting system 1441 consists of a (human) painting artist 1442 and acomputer 1443 that is interfaced to motion and action sensing devicesand painting-frame capture sensors to capture and record the artist'smovements and processes, and store in memory 1380 the associatedsoftware painting files. The commercial robotic painting system 1445 iscomprised of a user 1446 and a computer 1447 with a robotic paintingengine capable of interfacing and controlling robotic arms to recreatethe movements of the painting artist 1442 according to the softwarepainting files or applications along with visual feedback for thepurpose of calibrating a simulation model.

FIG. 95B depicts the robotic painting system architecture 1430. Thearchitecture includes a computer 1420, which is interfaced to/withmultiple external devices, including, but not limited to, motion sensinginput devices and touch-frame 1424, a standardized workstation 1425,including an easel 1426, a rinsing sink 1427, an art horse 1428, astorage cabinet 1429 and material containers 1430 (paint, solvents,etc.), as well as standardized tools and accessories (brushes, paints,etc.) 1431, visual input devices (camera, etc.) 1432, and one or morerobotic arms 1433.

The computer module 1420 includes modules that include, but are notlimited to, a robotic painting engine 1352 interfaced to a paintingmovement emulator 1422, a painting control module 1421 that acts basedon visual feedback of the painting execution processes, a memory module1380 to store painting execution program files, algorithms 1423 forlearning the selection and usage of the appropriate drawing tools, aswell as an extended simulation validation and calibration module 1378.

FIG. 95C depicts a robotic human-painting skill-replication engine 1352.In the replication engine 1352, there are multiple additional modulesall interconnected to each other over a common inter-modulecommunication bus 72. The replication engine contains further modules,including, but not limited to, an input module 1370, a paint movementrecording module 1372, an ancillary/additional sensory data recordingmodule 1376, a painting movement programming module 1374, a memorymodule 1380 containing software execution procedure program files, anexecution procedure module 1382 that generates execution commands basedon recorded sensor data, a module 1400 containing standardized paintingparameters, an output module 1388, and an (output) quality checkingmodule 1378, all overseen by a software maintenance module 1386.

One embodiment of the art platform standardization is defined asfollows. First, standardized position and orientation (xyz) of any kindof art tools (brushes, paints, canvas, etc.) in the art platform.Second, standardized operation volume dimensions and architecture ineach art platform. Third, standardized art tools set in each artplatform. Fourth, standardized robotic arms and hands with a library ofmanipulations in each art platform. Fifth, standardizedthree-dimensional vision devices for creating dynamic three-dimensionalvision data for painting recording and execution tracking and qualitycheck function in each art platform. Sixth, standardizedtype/producer/mark/of all using paints during particular paintingexecution. Seventh, standardized type/producer/mark/size of canvasduring particular painting execution.

One main purpose to have Standardized Art Platform is to achieve thesame result of the painting process (i.e., the same painting) executingby the original painter and afterward duplicated by robotic ArtPlatform. Several main points to emphasize in using the standardized ArtPlatform: (1) have the same timeline (same sequence of manipulations,same initial and ending time of each manipulation, same speed of movingobject between manipulations) of Painter and automatic roboticexecution; and (2) there are quality checks (3D vision, sensors) toavoid any fail result after each manipulation during the paintingprocess. Therefore the risk of not having the same result is reduced ifthe painting was done at the standardized art platform. If anon-standardized art platform is used, this will increase the risk ofnot having the same result (i.e. not the same painting) becauseadjustment algorithms may be required when the painting is not executedat not the same volume, with the same art tools, with the same paint orwith the same canvas in the painter studio as in the robotic artplatform.

FIG. 96A depicts the studio painting system and programcommercialization process 1450. A first step 1451 is for the humanpainting artist to make decisions pertaining to the artwork to becreated in the studio robotic painting system, which includes decidingon such topics as the subject, composition, media, tools and equipment,etc. The artist inputs all this data to the robotic painting engine instep 1452, after which in step 1453 the artist sets up the standardizedworkstation, tools and equipment and accessories and materials, as wellas the motion and visual input devices as required and spelled out inthe set-up procedure. The artist sets the starting point of the processand turns on the studio painting system in step 1454, after which theartist then begins step 1455 of actually painting. In step 1456 thestudio painting system records the motions and video of the artist'smovements in real time and in a known xyz coordinate frame during theentire painting process. The data collected in the painting studio isthen stored in step 1457, allowing the robotic painting engine togenerate a simulation program 1458 based on the stored movement andmedia data. The robotic painting program execution files or applicationsfor the produced painting are developed and integrated for use bydifferent operating systems and mobile systems and submitted toApp-stores or other marketplace locations for sale as a single-usepurchase or on a subscription basis.

FIG. 96B depicts the logical execution flow 1460 for the roboticpainting engine. As a first step the user selects a painting title instep 1461, with the input being received by the robotic painting enginein step 1462. The robotic painting engine uploads the painting executionprogram files in step 1463 into the onboard memory, and then proceeds tostep 1464, where it calculates the necessary tools and accessories. Achecking step 1465 provides the answers as to whether there is ashortage of tools or accessories and materials; should there be ashortage, the system sends an alert 1466 or a suggestion to the user foran ordering list or an alternate painting. In the case of no shortage,the engine confirms the selection in step 1467, allowing the user toproceed to step 1468, comprised of setting up the standardizedworkstation, motion and visual input devices using the step-by-stepinstruction contained within the painting execution program files. Oncecompleted, the robotic painting engine performs a check-up step 1469 toverify the proper setup; should it detect an error through step 1470,the system engine will send an error alert 1472 to the user and promptthe user to re-check the setup and correct any detected deficiencies. Ifthe check passes with no errors detected, the setup will be confirmed bythe engine in step 1471, allowing it to prompt the user in step 1473 toset the starting point and power on the replication and visual feedbackand control systems. In step 1474 the robotic arm(s) will execute thesteps specified in the painting execution program file, includingmovements, usage of tools and equipment at an identical pace asspecified by the painting program execution files. A visual feedbackstep 1475 monitors the execution of the painting replication processagainst the controlled parameter data that define a successful executionof the painting process and its outcomes. The robotic painting enginefurther takes the step 1476 of simulation model verification to increasethe fidelity of the replication process, with the goal of the entirereplication process to reach an identical final state as captured andsaved by the studio painting system. Once the painting is completed, anotification 1477 is sent to the user, including drying and curing timefor the applied materials (paint, paste, etc.)

FIG. 97A depicts a human musical-instrument skill-replication engine1354. In the replication engine 1354, there are multiple additionalmodules all interconnected to each other over a common inter-modulecommunication bus 72. The replication engine contains further modules,including, but not limited to, an audible (digital) audio input module1370, a human's musical instrument playing movement recording module1390, an ancillary/additional sensory data recording module 1376, amusical instrument playing movement programming module 1392, a memorymodule 1380 containing software execution procedure program files, anexecution procedure module 1382 that generates execution commands basedon recorded sensor data, a module 1394 containing standardized musicalinstrument playing parameters (e.g. pace, pressure, angles, etc.), anoutput module 1388, and an (output) quality checking module 1378, alloverseen by a software maintenance module 1386.

FIG. 96B depicts the process carried out and the logical flow for amusician replication engine 1480. To start, in step 1481 a user selectsa music title and/or composer, and is then queried in step 1482 whetherthe selection should be made by the robotic engine or throughinteraction with the human.

In the case the user selects the robot engine to select thetitle/composer in step 1482, the engine uses its own interpretation ofcreativity in step 1492, to offer the human user to provide input to theselection process in step 1493. Should the human decline providinginput, the robotic musician engine uses settings such as manual inputsto tonality, pitch and instrumentation as well as melodic variation instep 1499, to gather the necessary input in step 1130 to generate andupload selected instrument playing execution program files in step 1501,allowing the user to select the preferred one in step 1503, after therobotic musician engine has confirmed the selection in step 1502. Thechoice made by the human is then stored as a personal choice in thepersonal profile database in step 1504. Should the human decide toprovide input to the query in step 1493, the user will be able in step1493 to provide additional emotional input to the selection process(facial expressions, photo, news article, etc.). The input from step 194is received by the robotic musician engine in step 1495, allowing it toproceed to step 1496, where the engine carries out a sentiment analysisrelated to all available input data and uploads a music selection basedon the mood and style appropriate to the emotional input data from thehuman. Upon confirmation of selection for the uploaded music selectionin step 1497 by the robotic musician engine, the user may select the‘start’ button to play the program file for the selection in step 1498.

In the case where the human wants to be intimately involved in theselection of the title/composer, the system provides a list ofperformers for the selected title to the human on a display in step1483. In step 1484 the user selects the desired performer, a choiceinput that the system receives in step 1485. In step 1486 the roboticmusician engine generates and uploads the instrument playing executionprogram files, and proceeds in step 1487 to compare potentiallimitations between a human and a robotic musician's playing performanceon a particular instrument, thereby allowing it to calculate a potentialperformance gap. A checking step 1488 decides whether there exists agap. Should there be a gap, the system will suggest other selectionsbased on the user's preference profile in step 1489. Should there be noperformance gap, the robotic musician engine will confirm the selectionin step 1490 and allow the user to proceed to step 1491, where the usermay select the ‘start’ button to play the program file for theselection.

FIG. 98 depicts a human nursing-care skill-replication engine 1356. Inthe replication engine 1356, there are multiple additional modules allinterconnected to each other over a common inter-module communicationbus 72. The replication engine contains further modules, including, butnot limited to, an input module 1370, a nursing care movement recordingmodule 1396, an ancillary/additional sensory data recording module 1376,a nursing care movement programming module 1398, a memory module 1380containing software execution procedure program files, an executionprocedure module 1382 that generates execution commands based onrecorded sensor data, a module 1400 containing standardized nursing careparameters, an output module 1388, and an (output) quality checkingmodule 1378, all overseen by a software maintenance module 1386.

FIG. 99A depicts a robotic human nursing care system process 1132. Afirst step 1511 involves a user (care receiver or family/friends)creating an account for the care receiver, providing personal data(name, age, ID, etc.). A biometric data collection step 1512 involvesthe collection of personal data, including facial images, fingerprints,voice samples, etc. The user then enters contact information foremergency contact in step 1513. The robotic engine receives all thisinput data to build up a user account and profile in step 1514. Shouldthe user not be under a remote health monitoring program as determinedin step 1515, the robot engine sends an account creation confirmationmessage and a self-downloading manual file/app to the user's tablet, TV,smartphone or other device for future touch-screen or voice-basedcommand interface purposes, as part of step 1521. Should the user bepart of a remote health-monitoring program, the robot engine willrequest in step 1516 permission to access medical records. As part ofstep 1517 the robotic engine connects with the user's hospital andphysician's offices, laboratories and medical insurance databases toreceive the medical history, prescription, treatment, and appointmentsdata for the user and generates a medical care execution program forstorage in a file particular to that user. As a next step 1518, therobotic engine connects with any and all of the user's wearable medicaldevices (such as blood pressure monitors, pulse and blood-oxygensensors), or even electronically controllable drug dispensing system(whether oral or by injection) to allow for continuous monitoring. As afollow-on step the robotic engine receives medical data file and sensoryinputs allowing it to generate one or more medical care executionprogram files for the user's account in step 1519. The next step 1134involves the creation of a secure cloud storage data space for theuser's information, daily activities, associated parameters and any pastor future medical events or appointments. As before in step 1521, therobot engine sends an account creation confirmation message and aself-downloading manual file/app to the user's tablet, TV, smartphone orother device for future touch-screen or voice-based command interfacepurposes.

FIG. 99B depicts a continuation of the robotic human nursing care systemprocess 1132 first started with FIG. 99A, but which is now related to aphysically present robot in the user's environment. As a first step1522, the user turns on the robot in a default configuration andlocation (e.g. charging station). In task 1523 the robot receives auser's voice or touch-screen-based command to execute one specific orgroups of commands or actions. In step 1524, the robot carries outparticular tasks and activities based on engagement with the user usingvoice and facial recognition commands and cues, responses or behaviorsof the user, basing its decisions on such factors as task-urgency andtask-priority based on a knowledge of the particular or overallsituation. In task 1525 the robot carries out typical fetching, graspingand transportation of one or more items, completing the tasks usingobject recognition and environmental sensing, localization and mappingalgorithms to optimize movements along obstacle-free paths, possiblyeven to serve as an avatar to provide audio/video teleconferencingability for the user or interface with any controllable home appliance.The robot is continually monitoring the user's medical condition basedon sensory input and the user's profile data, and monitors for possiblesymptoms of potential medically dangerous conditions, with the abilityto inform first responders or family members about any potentialsituations requiring their immediate attention. The robot continuallychecks in step 1526 for any open or remaining task and always remainsready to react to any user input from step 1522.

FIG. 100 is a block diagram illustrating an example of a computerdevice, as shown in 224, on which computer-executable instructions toperform the methodologies discussed herein may be installed and run. Asalluded to above, the various computer-based devices discussed inconnection with the present invention may share similar attributes. Eachof the computer devices in 24 is capable of executing a set ofinstructions to cause the computer device to perform any one or more ofthe methodologies discussed herein. The computer devices 12 mayrepresent any or all of the 24, server 10, or any network intermediarydevices. Further, while only a single machine is illustrated, the term“machine” shall also be taken to include any collection of machines thatindividually or jointly execute a set (or multiple sets) of instructionsto perform any one or more of the methodologies discussed herein. Theexample computer system 224 includes a processor 226 (e.g., a centralprocessing unit (CPU), a graphics processing unit (GPU), or both), amain memory 228 and a static memory 230, which communicate with eachother via a bus 232. The computer system 224 may further include a videodisplay unit 234 (e.g., a liquid crystal display (LCD)). The computersystem 224 also includes an alphanumeric input device 236 (e.g., akeyboard), a cursor control device 238 (e.g., a mouse), a disk driveunit 240, a signal generation device 242 (e.g., a speaker), and anetwork interface device 248.

The disk drive unit 2240 includes a machine-readable medium 244 on whichis stored one or more sets of instructions (e.g., software 246)embodying any one or more of the methodologies or functions describedherein. The software 246 may also reside, completely or at leastpartially, within the main memory 244 and/or within the processor 226during execution thereof the computer system 224, the main memory 228,and the instruction-storing portions of processor 226 constitutingmachine-readable media. The software 246 may further be transmitted orreceived over a network 18 via the network interface device 248.

While the machine-readable medium 244 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any tangible medium that iscapable of storing a set of instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present invention. The term “machine-readablemedium” shall accordingly be taken to include, but not be limited to,solid-state memories, and optical and magnetic media.

In general terms, there may be considered a method of motion capture andanalysis for a robotics system, comprising sensing a sequence ofobservations of a person's movements by a plurality of robotic sensorsas the person prepares a product using working equipment; detecting inthe sequence of observations mini-manipulations corresponding to asequence of movements carried out in each stage of preparing theproduct; transforming the sensed sequence of observations into computerreadable instructions for controlling a robotic apparatus capable ofperforming the sequences of mini-manipulations; storing at least thesequence of instructions for mini-manipulations to electronic media forthe product. This may be repeated for multiple products. The sequence ofmini-manipulations for the product is preferably stored as an electronicrecord. The mini-manipulations may be abstracted parts of a multi-stageprocess, such as cutting an object, heating an object (in an oven or ona stove with oil or water), or similar. Then, the method may furthercomprise: transmitting the electronic record for the product to arobotic apparatus capable of replicating the sequence of storedmini-manipulations, corresponding to the original actions of the person.Moreover, the method may further comprise executing the sequence ofinstructions for mini-manipulations for the product by the roboticapparatus, thereby obtaining substantially the same result as theoriginal product prepared by the person.

In another general aspect, there may be considered a method of operatinga robotics apparatus, comprising providing a sequence of pre-programmedinstructions for standard mini-manipulations, wherein eachmini-manipulation produces at least one identifiable result in a stageof preparing a product; sensing a sequence of observations correspondingto a person's movements by a plurality of robotic sensors as the personprepares the product using equipment; detecting standardmini-manipulations in the sequence of observations, wherein amini-manipulation corresponds to one or more observations, and thesequence of mini-manipulations corresponds to the preparation of theproduct; transforming the sequence of observations into roboticinstructions based on software implemented methods for recognizingsequences of pre-programmed standard mini-manipulations based on thesensed sequence of person motions, the mini-manipulations eachcomprising a sequence of robotic instructions and the roboticinstructions including dynamic sensing operations and robotic actionoperations; storing the sequence of mini-manipulations and theircorresponding robotic instructions in electronic media. Preferably, thesequence of instructions and corresponding mini-manipulations for theproduct are stored as an electronic record for preparing the product.This may be repeated for multiple products. The method may furtherinclude transmitting the sequence of instructions (preferably in theform of the electronic record) to a robotics apparatus capable ofreplicating and executing the sequence of robotic instructions. Themethod may further comprise executing the robotic instructions for theproduct by the robotics apparatus, thereby obtaining substantially thesame result as the original product prepared by the human. Where themethod is repeated for multiple products, the method may additionallycomprise providing a library of electronic descriptions of one or moreproducts, including the name of the product, ingredients of the productand the method (such as a recipe) for making the product fromingredients.

Another generalized aspect provides a method of operating a roboticsapparatus comprising receiving an instruction set for a making a productcomprising of a series of indications of mini-manipulationscorresponding to original actions of a person, each indicationcomprising a sequence of robotic instructions and the roboticinstructions including dynamic sensing operations and robotic actionoperations; providing the instruction set to a robotic apparatus capableof replicating the sequence of mini-manipulations; executing thesequence of instructions for mini-manipulations for the product by therobotic apparatus, thereby obtaining substantially the same result asthe original product prepared by the person.

A further generalized method of operating a robotic apparatus may beconsidered in a different aspect, comprising executing a roboticinstructions script for duplicating a recipe having a plurality ofproduct preparation movements; determining if each preparation movementis identified as a standard grabbing action of a standard tool or astandard object, a standard hand-manipulation action or object, or anon-standard object; and for each preparation movement, one or more of:instructing the robotic cooking device to access a first databaselibrary if the preparation movement involves a standard grabbing actionof a standard object; instructing the robotic cooking device to access asecond database library if the food preparation movement involves astandard hand-manipulation action or object; and instructing the roboticcooking device to create a three-dimensional model of the non-standardobject if the food preparation movement involves a non-standard object.The determining and/or instructing steps may be particularly implementedat or by a computer system. The computing system may have a processorand memory.

Another aspect may be found in a method for product preparation byrobotic apparatus, comprising replicating a recipe by preparing aproduct (such as a food dish) via the robotic apparatus, the recipedecomposed into one or more preparation stages, each preparation stagedecomposed into a sequence of mini-manipulations and active primitives,each mini-manipulation decomposed into a sequence of action primitives.Preferably, each mini manipulation has been (successfully) tested toproduce an optimal result for that mini manipulation in view of anyvariations in positions, orientations, shapes of an applicable object,and one or more applicable ingredients.

A further method aspect may be considered in a method for recipe scriptgeneration, comprising receiving filtered raw data from sensors in thesurroundings of a standardized working environment module, such as akitchen environment; generating a sequence of script data from thefiltered raw data; and transforming the sequence of script data intomachine-readable and machine-executable commands for preparing aproduct, the machine-readable and machine-executable commands includingcommands for controlling a pair of robotic arms and hands to perform afunction. The function may be from the group consisting of one or morecooking stages, one or more mini-manipulations, and one or more actionprimitives. A recipe script generation system comprising hardware and/orsoftware features configured to operate in accordance with this methodmay also be considered.

In any of these aspects, the following may be considered. Thepreparation of the product normally uses ingredients. Executing theinstructions typically includes sensing properties of the ingredientsused in preparing the product. The product may be a food dish inaccordance with a (food) recipe (which may be held in an electronicdescription) and the person may be a chef. The working equipment maycomprise kitchen equipment. These methods may be used in combinationwith any one or more of the other features described herein. One, morethan one or all of the features of the aspects may be combined, so afeature from one aspect may be combined with another aspect for example.Each aspect may be computer-implemented and there may be provided acomputer program configured to perform each method when operated by acomputer or processor. Each computer program may be stored on acomputer-readable medium. Additionally or alternatively, the programsmay be partially or fully hardware-implemented. The aspects may becombined. There may also be provided a robotics system configured tooperate in accordance with the method described in respect of any ofthese aspects.

In another aspect, there may be provided a robotics system, comprising:a multi-modal sensing system capable of observing human motions andgenerating human motions data in a first instrumented environment; and aprocessor (which may be a computer), communicatively coupled to themulti-modal sensing system, for recording the human motions datareceived from the multi-modal sensing system and processing the humanmotions data to extract motion primitives, preferably such that themotion primitives define operations of a robotics system. The motionprimitives may be mini-manipulations, as described herein (for examplein the immediately preceding paragraphs) and may have a standard format.The motion primitive may define specific types of action and parametersof the type of action, for example a pulling action with a definedstarting point, end point, force and grip type. Optionally, there may befurther provided a robotics apparatus, communicatively coupled to theprocessor and/or multi-modal sensing system. The robotics apparatus maybe capable of using the motion primitives and/or the human motions datato replicate the observed human motions in a second instrumentedenvironment.

In a further aspect, there may provided a robotics system, comprising: aprocessor (which may be a computer), for receiving motion primitivesdefining operations of a robotics system, the motion primitives beingbased on human motions data captured from human motions; and a roboticssystem, communicatively coupled to the processor, capable of using themotion primitives to replicate human motions in an instrumentedenvironment. It will be understood that these aspects may be furthercombined.

A further aspect may be found in a robotics system comprising: first andsecond robotic arms; first and second robotic hands, each hand having awrist coupled to a respective arm, each hand having a palm and multiplearticulated fingers, each articulated finger on the respective handhaving at least one sensor; and first and second gloves, each glovecovering the respective hand having a plurality of embedded sensors.Preferably the robotics system is a robotic kitchen system.

There may further be provided, in a different but related aspect, amotion capture system, comprising: a standardized working environmentmodule, preferably a kitchen; plurality of multi-modal sensors having afirst type of sensors configured to be physically coupled to a human anda second type of sensors configured to be spaced away from the human.One or more of the following may be the case: the first type of sensorsmay be for measuring the posture of human appendages and sensing motiondata of the human appendages; the second type of sensors may be fordetermining a spatial registration of the three-dimensionalconfigurations of one or more of the environment, objects, movements,and locations of human appendages; the second type of sensors may beconfigured to sense activity data; the standardized working environmentmay have connectors to interface with the second type of sensors; thefirst type of sensors and the second type of sensors measure motion dataand activity data, and send both the motion data and the activity datato a computer for storage and processing for product (such as food)preparation.

An aspect may additionally or alternatively be considered in a robotichand coated with a sensing gloves, comprising: five fingers; and a palmconnected to the five fingers, the palm having internal joints and adeformable surface material in three regions; a first deformable regiondisposed on a radial side of the palm and near the base of the thumb; asecond deformable region disposed on a ulnar side of the palm, andspaced apart from the radial side; and a third deformable regiondisposed on the palm and extend across the base of the fingers.Preferably, the combination of the first deformable region, the seconddeformable region, the third deformable region, and the internal jointscollectively operate to perform a mini manipulation, particularly forfood preparation.

In respect of any of the above system, device or apparatus aspects,there may further be provided method aspects comprising steps to carryout the functionality of the system. Additionally or alternatively,optional features may be found based on any one or more of the featuresdescribed herein with respect to other aspects

TABLE A Types of Equipment Types of Equipment KITCHEN ACCESSORIES 1Funnels 1.1. stainless steel funnel 1.2. plastic funnel 1.3 siliconefunnel 1.4 convertible funnel 2 Colanders 2.1 quadratic colanders 2.2oval ladle-vases 2.3 colanders with folding handles 2.4 flat colander2.5 plastic colanders 2.6 small round colanders 2.7 suspended colanders2.8 cover-colander 2.9 stainless steel and aluminum colanders 2.1 conecolanders 3 Kitchen Appliances 3.1. Whisk 3.2. scoop, spatula 3.2.1 cookspatula 3.2.2. spatula with slots 3.2.3. confectionery spatula 3.5Spoons 3.5.1 serving spoon 3.5.2 spoon-tongs 3.5.3 spoon with slots3.5.4 spoon for rice 3.5.5 ladle spoon 3.5.6 ice cream spoon 3.5.7 honeyspoon 3.5.8 spaghetti spoon 3.5.9 serving spoon 3.6 confectionerysyringe for cookies and cream 3.7 soup ladle 3.8 Potato Masher 3.9skimmer 3.10 Meat fork 3.11 Brush 3.12 coffee filter 3.5.7 honey spoon3.5.8 spaghetti spoon 3.5.9 serving spoon 3.6 confectionery syringe forcookies and cream 3.7 soup ladle 3.8 Potato Masher 3.9 skimmer 3.10 Meatfork 3.11 Brush 3.23 ties for rolls 3.2 dough mini-scraper 3.25 grilltongs 3.26 spaghetti tongs 3.27 ice tongs 3.28 sugar tongs 3.29 packageclip 3.30 package clip 3.31 citrus spray 3.32 Dough press 3.33 scoop forbulk 3.34 salad serving tongs (tweezers) 3.35 accessories for tubes 3.36Pestle 3.37 Mortar 3.38 roller for cutting of the rings 3.39 opener forcaps 3.40 meat tenderizer; meat softener 3.41 egg yolk separator 3.42Apron 3.33 scoop for bulk 3.43 tools for decoration 3.44 jar for oil andvinegar 3.45 mug for milk boiling 3.46 napkins 3.47 tablecloth 3.48marker for glasses 3.49 potato masher 3.50 Basket 3.51 meat tenderizer3.52 cocotte 3.53 brush for washing of the vegetables 3.54 lids for cups3.55 rope for baking 3.56 jar for herbs storage 3.57 Mortar 3.58 scraperfor glass ceramic plates 3.59 Teapot for tea 3.60 clothespin for noteson the fridge 3.61 railing systems 3.62 hanger for kitchen tools 3.63plunger with not adhering surface 3.64 silicone plunger 3.65 rolling pinwith adjustable thickness 3.66 vacuum bags with pump 3.67 gas lighter3.68 bone forceps 4 kitchen timers, thermometers 4.1 timer for meatroasting 4.2 digital thermometer 4.3 holder for thermometer 4.4 meatthermometer 4.5 digital timer 4.6 elector. digital timer 4.7 _aramelthermometer 5 Mills for spices 5.1 mill for black pepper 5.2 electricmill 5.3 combined mill for pepper and salt (2 in 1) 5.4 mill for spices5.5 mill for greens 6 Measuring utensils 6.1. Measuring container(plastic bottle) 6.2. measuring jar 6.3. measuring jug 6.4. measuringbowl 6.5. mechanical dispenser for ice cream 7 mechanical mixers 8 Bowl8.1. metal bowl 8.2. stainless steel bowl 8.3. plastic bowl 8.4. plasticbowl 8.5. bowls for food 9 Sets 9.2 wine set 9.3 sets for spices 9.6.cupcakes baking set 9.7 accessory kit for baking 9.8 set of bar tools9.9 set of kitchen tools 9.10 Set for eggs and pancakes baking 11Slicing and cutting of products 11.1 Cutter 11.2 holder for onionscutting 11.2 cutting boards 11.3 universal professional knives 11.4kitchen shears 11.5 hatchet 11.6 meat hatchet 11.7 Hammer for meat withhatchet 11.8 Hoe 11.9 Hammer for meat 11.10 Knives 11.11 knife forgreens 11.12 knife for oranges 11.13 knife for kiwi 11.14 knife forpineapple 11.15 Spiral knife for carrots 11.16 multifunctional knife11.17 vegetable knife 11.18 Pizza Cutter 11.19 universal knife 11.20knife for slicing 11.21 cook knife 11.22 gastronomic knife 11.23 opener11.24 Cheese knife 11.25 boning knife 11.26 lettuce knife 11.27 knifefor steaks 11.28 butcher knife 11.29 shredding knife 11.30 bread knife11.31 fish knife 11.32 knife for sandwiches 11.33 Santoku knife 11.34knife for fruit coring 11.35 Butter knife 12 openers 12.1 tin-opener12.2 corkscrew 12.3 corkscrew on a stand 12.4 lever corkscrew 12.5folding corkscrew 12.6 opener for waiter 12.7 openers 13 stand andholders 13.1 stands for hot 13.2 stand for kitchen utensils storing 13.3toothpick holder 13.4 Bottle holder 13.5 Holder for capsules 13.6 standfor spoon 13.7 stand for coffee capsules 13.8 Coasters 13.9 Napkinholder 13.10 stand for eggs 13.11 stand for openers 13.12 stand forscoops 13.13 stand for cooking and serving of eggs 13.14 stand for ladle13.15 Holder for paper towels 13.16 Transforming stand for kitchenappliances 13.17 stand for mug 13.18 stand mugs and saucers 13.19 standfor kitchen knives 13.20 stand for chicken 13.21 napkin-stand 13.22heated stand 13.23 stands for cake 14 Appliances for peeling and cutting14.1 grater for vegetables 14.2 grater 14.3 garlic masher 14.4 eggcutter 14.5 Manual vegetable cutter 14.6 Peeler for vegetables 14.7Nutcracker 14.8 The device for separating the yolks from the whites 14.9grasping for carrots cleaning 14.10 scraper fish scales 14.11 cutter forfruits 14.12 roller for holes 14.13 tongs for fish bones 14.14 spiralvegetable cutter 15 Bottle Caps 15.1 champagne cork (stopper) 15.2stoppers for wine 15.3 The opener to remove the crown corks from bottles16 sieves 16.1 sieve for tea 16.2 sieve-tongs for tea 16.3 Strainer forspices 16.4 Strainer for tea 16.5 Universal sieve 16.6 flour sieve 16.7sieve to form the “Bird's Nest” 16.8 The Chinese sieve with a meshinsert 16.9 sieve with support 16.10 Mug-sieve for flour 16.11 sieve onthe handle 17 Salt and pepper shakers 17.1 container for seasoning 17.2salt cellar 17.3 containers for oil and vinegar 18 Dish dryers 18.1salad dryer dryer-placemat dryer for crockery and cutlery 19 CutleryAccessories 19.1 cutlery tray 19.2 cutlery holder 19.3 cutlery container19.4 strainer for cutlery 19.5 wall hanger for kitchen tools 19.6cutlery organizer 19.7 mat for cutlery 19.8 sliding tray for cutlery19.9 dryer for cutlery 19.10 glass for cutlery 19.11 napkin for thecutlery 19.12 case for cutlery 19.13 tray for cutlery 19.14mitten-potholder 19.15 box for cutlery 19.16 full-size rack (cassette)for cutlery 19.17 Stand without containers for cutlery 19.18 cassettefor cutlery 19.19 container for cutlery 19.20 station for cutlery 19.21Shelf for cutlery 20 Decorations for cocktails 21.1 Ducts 22.2. Sticks23 Mold 23.1 molds for ice 23.2 molds for children 23.3 Molds forshaping products 23.4 Molds for dumplings 24 Measuring container 24.1Measuring container 24.2 A mixing container with the dispenser 24.3Measuring container with the funnel 24.4 Beaker 24.5 Scoop 26 kitchenscissors 26.1 Scissors for BBQ 26.2 Kitchen scissors with bottle opener26.3 Scissors for greens 26.4 Kitchen multipurpose scissors 26.5 Kitchenscissors for poultry 27 utensil for storage 27.1 container for storage27.2 Bottles for liquid spices, oils 27.3 jars for storage 27.4 lunchbox27.5 foldable lunchbox 27.6 jar for hermetic storage of bulk products27.7 Sprayer for oil/vinegar 27.8 jar for bulk products 27.9 containersfor spices 27.10 container for seasoning 27.11 Container for tea 28potholders 28.1 oven-glove 28.2 silicone potholders 28.3 dishclothrailing with hooks 29 silicone mats 29.1 baking mat 29.2 mat for bakingcakes 29.3 mat for drying of the glasses 29.4 cooking mat 29.5 Mat fordrying of the dishes 30 graters, presses, rubbing machines 30.1 graterwith a handle 30.2 grater 30.3 multifunction grater 30.4 grater shredder30.5 grind for the green 30.6 grind for the garlic 30.7 Slicer fortomatoes 30.8 grater with rotating drums 30.9 universal device forgrinding 30.10 mechanical grater 30.11 garlic peeling tube 30.12 rubbingmachine 30.13 press for vegetables 30.14 press for garlic 30.2 press forhamburgers 31 knife sharpener 31.1 electric sharpener 31.2 sharpeningstone 31.3 ceramic sharpener 32 breadbox 33 lattice with legs Kitchendishes 1 for alcohol 1.2 Brandy set with dispenser 1.3 souvenir cups 1.4stemware 1.5 pail of ice 1.6 stemware 1.7 champagne bucket 1.8 stemware1.9 carafe 1.10 server 1.11 bottle holder 2 tableware 2.1 first coursedish 2.2 dish for bouillon 2.3 bouillon bowl 2.4 oiler 2.5 round dish2.6 duck pan 2.7 Set for making chocolate fondue 2.8 Set for makingcheese fondue 2.9 salad bowl 2.10 dish for cake 2.11 compartmental dish2.12 set of cutlery 2.13 serving spoon and fork 2.14 dish with lid 2.15steam table 2.16 ice-cream bowl 2.17 Flatware 2.18 saucer 2.19 saucerfor jam 2.20 mustard-pot 2.21 pepper-pot 2.22 ash-pot 2.23 deep tableplate 2.24 dinner plate 2.25 snack plate 2.26 deep dessert plate 2.27dessert plate 2.28 plate for pies 2.29 horseradish-pot 3 Utensils fortable 3.1 Pad for tableware 3.2 serving mat 3.3. serving tray 3.4 glassburner 4 Dishes for tea, coffee, dessert 4.1 sugar-bowl 4.2 mug 4.3 mugwith teapot 4.4 mug with stand 4.5 mug with lid 4.6 tea set 4.7 dish 4.8french-press 4.9 teapot 4.10 teapot with strainer 4.11 glass teapot 4.12ice-cream bowl 4.13 multifunctional vase 4.14 glasses 4.15 soup bowl4.16 wicker basket 4.17 vase 3-tier 4.18 tea set 4.19 napkin rings 4.20pannier for fruits 4.21 table trash basket 4.22 biscuit dish 4.23 candydish 4.24 coffee sets 5 CUTLERY 5.1 Table fork 5.2 fork for sprat 5.3fork for crayfish 5.4 fork for oysters 5.6 fork for lemons 5.7 big spoon5.8 dessert spoon 5.9 tea spoon 5.10 coffee spoon 5.11 lemonade spoon5.12 ladle-spoon 5.13 spoon for hot snacks 5.14 ice cream spoon 5.15mustard spoon 5.16 salt spoon 5.17 spatula for cakes 5.18 spatula forcaviar 5.19 spatula for fish 5.20 table knife 5.21 knife and fork forthe fish 5.22 knife and fork snack 5.23 knife and fork dessert 5.24Butterknife 5.25 tool kits for lobster, crayfish 5.26 devices for spices5.27 grille and asparagus tongs 5.28 salad unit (salad fork and spoon)5.29 sugar-tongs 5.30 tongs for cakes and sugar 5.31 ice tongs 5.32can-opener 5.33 fork oyster 5.34 plug for crayfish 5.36 cocotte fork5.37 fork for canned fish in oil (sprat, sardines) 5.38 spinner forchampagne 5.39 spoon to mix whiskey with soda water Kitchen appliances 1aerogrill 2 blenders, grinder 3 coffee Maker 4 coffee grinder (coffeemill) 5 Food Processor 6 mixer 7 mini oven 8 multicooker 9 meat grinders10 steamers 11 Raclette grill 12 Juicers 13 toasters 14 egg cooker 15electric range 15.1 electric induction stove 16 electric kettle 16.1thermopots 17 bread makers 18 microwaves 19 weights for kitchen 20electric driers 21 weights for kitchen Children's dishes 1 Children Setsfor baking 2 Children cutlery 3 Children thermoses 4 Children Sets ofdishes List of ingredient data 1 Ingredient name 2 Ingredient Photo 3Manufacturer 4 Country 5 Type of Ingredient 6 Type of cuisine 7 Relationto Vegetarianism 8 Spice 9 Energy value 10 Description of the Ingredient11 Status 12 Price List of equipment data 1 Equipment name 2 Equipmentphoto 3 Manufacturer 4 Brand name 5 Dimensions 6 Weight 7 Connectivity 8Type of cuisine 9 Type of equipment 10 Description of equipment 11 Year12 Status 13 Price List of recipe data 1 Name of the recipe 2 Recipeauthor 3 Recipe Photo 4 Preparation time 5 Basis of the dish 6 Type ofcuisine 7 Type of the dish 8 Relation to Vegetarianism 9 Spice 10 Energyvalue 11 Number of persons 12 Description of the recipe 13 Descriptionof the stages of cooking 14 Ingredients 15 Type of equipment 16 Video ofrecipe cooking 17 User Rating 18 Expert Rating 19 Amount of sales 21Automatic cooking 22 Price

TABLE B Types of Ingredients 1 MEAT and MEAT PRODUCTS 1 Basturma 2 Fat 3brisket cooked and smoked 4 Hare 5 leather duck 6 Sausage 7 Sausages 8Sausages “Hunting party” 9 Horsemeat 10 Bones with bone marrow 11 Roe 12Rabbit 13 Meat 14 Moosemeat 15 Venison 16 Liver 17 Kidney 18 Smoked ribs19 Salami 20 Sausages 21 Cervelat 22 Sausages 23 Hungarian smoked bacon24 bacon fat-tailed 25 Steak 26 ribeye steak 27 Farce 28 crocodilefillet 29 Jamon 30 Choriso (spanish sausage) 31 Skewers 32 Sowbelly 33Deer tongue 34 Frog legs LAMB, VEAL 1 breast of lamb 2 loin of lamb 3blade lamb 4 veal brains 5 mutton ham 6 veal ham 7 leg of lamb 8 Heelmuscle mutton 9 lamb offal 10 veal kidneys 11 lamb chops 12 gras cow 13veal heart 14 lamb testicles 15 veal fillet 16 veal cheeks 17 mincedlamb 18 minced veal 19 veal tail 20 veal tongue 21 eggs bullish BEEF 1beef brisket 2 beef fillet 3 beef (sirloin) 4 beef on the bone 5 beefeye muscle 6 legs of beef 7 ham beef 8 gras beef 9 beef ribs 10 beefheart 11 minced beef 12 tail beef 13 beef tongue PORK 1 bacon 2 smokedbacon 3 Pork 4 Ham 5 pork brisket 6 smoked pork belly 7 Pork Intestine 8legs of pork 9 boar ham 10 pork ham 11 pork ribs 12 knuckle of pork 13Fat 14 pork (pork neck or loin) 15 pork ears 16 minced pork 17 pig tail18 pork tongue 2 BIRDS 1 garshnep 2 turkey breast 3 chicken breast 4chicken breast, smoked 5 duck breast 6 Goose 7 chicken ventricles 8turkey 9 turkey wings 10 chicken wings 11 chicken 12 smoked chicken 13grouse 14 Coot 15 duck leg 16 crow's feet 17 chicken legs 18 chicken ham19 Quail 20 gras chicken 21 chicken giblets 22 grouse 23 chicken hearts24 Duck 25 smoked duck 26 Pheasant 27 minced chicken 28 chicken fillet29 foie gras 30 chicken 31 chicken gutted 32 neck duck 3 FISH andSEAFOOD 1 anchovies 2 arctic char 3 mullet 4 Black Sea goby 5 shrimphead 6 Butterfish 7 scallops 8 dorado 9 Ruff 10 caviar 11 red caviar 12Tobiko caviar 13 Squid 14 flounder 15 cuttlefish 16 Carp 17 Sprat 18Smelt 19 crab sticks 20 Shrimps 21 King shrimps 22 Salad shrimps 23Tiger prawn 24 Bream 25 salmon 26 Smoked salmon 27 Mussels 28 Musselswith shells 29 Pollock 30 Molluscs 31 Sea food 32 Sea fish 33 sole(fish) 34 Crab meat 35 Krill meat 36 Burbot 37 Perch 38 Lobster 39 Cisco40 sturgeon 41 octopus 42 baby octopus 43 shrimp broth 44 halibut 45Pangasius 46 cod liver oil 47 Haddock 48 Crayfish 49 dried crustaceans50 Hot smoked fish 51 red fish salted 52 Swordfish 53 Saury 54 Sardines55 Herring 56 Salmon 57 smoked salmon 58 salted salmon 59 Seabass 60Whitefish 61 Ramp 62 Mackerel 63 smoked mackerel 64 Sheatfish 65 Starlet66 Walleye 67 Dried seaweed 68 Tilapia 69 Carp 70 Cod 71 Hot smoked cod72 black cod 73 Tuna 74 Turbot 75 Eel 76 smoked eel 77 Snails 78 Oysters79 white fish fillets 80 catfish fillets 81 fillet of carp 82 fishfillet 83 salmon fillet 84 salted herring fillets 85 perch fillet 86Trout 87 smoked trout 88 Squid Ink 89 cervical shrimp 90 cervicalcancers 91 Sprats 92 Pike 4 VEGETABLES 2 Artichokes 3 Eggplant 4 Yam 5broccoli tops 6 beet tops 7 Broccoli 8 Rutabaga 9 Galangal 10 Peas 11pea sprouts 12 pea pods 13 green peas 14 Daikon 15 Melon 16 Ginseng 17Ginger 18 Zucchini 19 Feces 20 Cabbage 21 Brussels sprouts 22 Sauerkraut23 Chinese cabbage 24 Cabbage 25 Romanesco cabbage 26 savoy cabbage 27Cauliflower 28 Potatoes 29 young potatoes 30 Kohlrabi 31 root anise 32salsify root 33 parsley root 34 celery root 35 fresh corn 36 white onion37 pearl bow 38 onion 39 red onion 40 dry onion 41 small onion 42Shallots 43 cassava 44 mini corn 45 mini peppers 46 mini-tomatoes 47carrots 48 cucumber 49 parsnips 50 squash 51 bell peppers 52 cayennepepper 53 fresh chili pepper 54 jalapeno peppers 55 tomato 56 pickledtomatoes 57 cherry tomatoes 58 sunflower sprouts 59 wheat germ 60soybean seedlings 61 germinated soybeans 62 rhubarb 63 Radish 64 wildradish 65 Turnip 66 beansprouts 67 Beet 68 Asparagus 69 chopped tomatoes70 Sweet 71 Pumpkin 72 green beans 73 Fennel 74 physalis 75 horseradish76 zucchini 77 Garlic 78 endive 5 FRUITS 1 Apricot 2 Avocado 3 quince 4fresh pineapple 5 Orange 6 banana 7 Hawthorn 8 cranberries 9 grapes 10Cherry 11 Dried cherries 12 blueberries 13 Garnet 14 Grapefruit 15 Pear16 Blackberry 17 strawberries 18 pomegranate seeds 19 carambola 20 Kiwi21 Strawberry 22 Cranberry 23 coconut 24 gooseberry 25 kumquat 26 Lime27 lemon 28 Litchi 29 raspberries 30 mango 31 Mandarin 32 Passionfruit33 mini pineapple 34 Nectarine 35 buckthorn 36 papaya 37 Peach 38 Pomelo39 Rowan 40 Drain 41 red currants 42 black currant 43 tamarind 44 Feijoa45 fruit to taste 46 persimmon 47 cherries 48 Cherry 49 blueberries 50Apple 51 frozen berries 52 juniper berries 53 fresh berries 6 GROCERY 1Agar 2 Adjika 3 rice paper 4 vanilla extract 5 vermicelli rice 6 eggnoodles 7 Algae 8 Glucose 9 Jam 10 raspberry jam 11 fresh yeast 12Gelatin 13 liquid Smokehouse 14 Sweetener 15 corn muffins 16 Ketchup 17citric acid 18 Candy 19 Confiture 20 strawberry jam 21 food dye 22Starch 23 potato starch 24 corn starch 25 bread crumbs 26 Noodles 27buckwheat noodles 28 Pad Thai noodles 29 rice noodles 30 glass noodles31 noodles harusame 32 egg noodles 33 Mayonnaise 34 poppy sweet 35 Pasta36 cannelloni pasta 37 pasta lumakoni 38 pasta feathers 39 fusilli pasta40 pumpkin marmalade 41 jujube fruit 42 Marzipan 43 Mirin 44 coconutmilk 45 almond milk 46 soy milk 47 Muesli 48 Pasta 49 peanut paste 50red curry paste 51 tamarind paste 52 Tom Yam Paste 53 chili paste 54Molasses 55 Pectin 56 Penne 57 Jam 58 elderberry syrup 59 vanilla syrup60 syrup vishnevny 61 ginger syrup 62 caramel syrup 63 maple syrup 64strawberry syrup 65 coffee syrup 66 corn syrup 67 raspberry syrup 68mango syrup 69 honey syrup 70 almond syrup 71 walnut syrup 72blackcurrant syrup 73 chocolate syrup 74 cranberry sauce 75worcestershire sauce 76 pomegranate sauce 77 kimchi sauce 78 Pesto 79fish sauce 80 fish sauce nampla 81 Tabasco sauce 82 teriyaki sauce 83sauce tkemali 84 oyster sauce 85 sweet chili sauce 86 Japanese walnutsauce 87 spaghetti 88 crumbs of white bread 89 breadcrumbs 90 pastrydecorations 91 candied 7 MILK PRODUCTS and EGGS 1 yogurt 2 naturalyoghurt 3 Kefir 4 margarine 5 butter 6 melted butter 7 Milk 8 baked milk9 buttermilk 10 curdled 11 cream 12 sour cream 13 Whey 14 Thane 15 Curd16 curd beaded 17 quail eggs 18 Egg 8 MUSHROOMS 2 mushrooms 3 Ceps 4Enoki mushrooms 5 Chinese dried mushrooms 6 portobello mushrooms 7 driedmushrooms 8 shiitake mushrooms 9 milkmushrooms 10 chanterelles 11boletus 12 honey fungus 13 saffron milk cap 14 morels 15 truffles 16meadow mushrooms 9 CHEESE 1 cheese 2 cheese Adyghe 3 brie cheese 4 fetacheese 5 Burrata cheese 6 Gouda cheese 7 Dutch cheese 8 blue cheese 9Gorgonzola 10 grana padano cheese 11 Gruyere cheese 12 Dor Blue cheese13 Camembert 14 goat cheese 15 cheese sausage 16 mascarpone cheese 17Monterey Jack cheese 18 mozzarella cheese 19 soft cheese 20 goat cheese21 parmesan cheese 22 pecorino cheese 23 processed cheese 24 cheesePoshehonsky 25 ricotta cheese 26 Roquefort cheese 27 blue cheese 28cream cheese 29 suluguni 30 cheese curd 31 feta cheese 32 philadelphiacheese 33 cheddar cheese 34 edam cheese 35 Emmentaler cheese 10 NUTS andDRIED FRUITS 1 peanuts 2 barberry 3 walnuts (peeled) 4 raisins 5 Figs 6Chestnut 7 Dried cranberries 8 coconut 9 dried apricots 10 Filbert(hazelnut) 11 almonds 12 Nuts 13 pine nuts 14 cashew nuts 15 Driedpeaches 16 sunflower seeds 17 pumpkin seeds 18 Dried Fruits 19 Dates 20Pistachios 21 Hazelnuts 22 Prunes 11 BEVERAGES 1 Water 2 water orange 3mineral water 4 water pink 5 GABA-tea 6 Hibiscus 7 Kvass 8 bread kvass 9Coke 10 Kuding 11 Lemonade 12 Mate 13 Juice 14 carbonated drink 15Bitter Brandy 16 Rooibos 17 pineapple juice 18 orange juice 19 birchjuice 20 grape juice 21 cherry juice 22 pomegranate juice 23 strawberryjuice 24 cranberry juice 25 gooseberry juice 26 lime juice 27 mangojuice 28 tangerine juice 29 peach juice 30 currant juice 31 tomato juice32 apple juice 33 Sprite 34 Tonic 35 tea white 36 tea yellow 37 greentea 38 red tea 39 Puer tea 40 Puer tea in Mandarin 41 oolong tea 42black tea 43 Espresso 12 ALCOHOL 1 Balm 2 Bitter 3 Brandy 4 Bourbon 5Vermouth 6 Wine 7 white wine 8 sparkling wine 9 red wine 10 dry red wine11 wine sangria 12 Whiskey 13 Vodka 14 anise vodka 15 Grappa 16 Gin 17Irish cream liqueur 18 Calvados 19 Cachaca 20 Brandy 21 Liqueur 22orange liqueur 23 coffee liqueur 24 chocolate liqueur 25 Madeira 26Marsala 27 Martini 28 Beer 29 cherry beer 30 Port 31 Rum 32 white rum 33black rum 34 Sake 35 sambuca 36 Cider 37 tequila 38 sherry 39 Champagne(Brut) 40 schnapps 13 GREENS AND HERBS 1 Basil 2 basil red 3 bouquetgarni 4 oregano 5 greens 6 dried herbs 7 cabbage pak choi 8 chervil 9cilantro 10 oxalis 11 oat root 12 fresh coriander 13 nettle 14Watercress 15 watercress 16 rose petals 17 lemongrass 18 bamboo leaves19 banana leaves 20 grape leaves 21 Grape leaves (salty) 22 kaffir limeleaves 23 lime leaves 24 dandelion leaves 25 green onion 26 Leek 27marjoram 28 Chard 29 melissa 30 lemon balm 31 Mint 32 oregano 33 parsley34 dried parsley 35 plantain 36 wormwood 37 chopped camomile 38 arugula39 iceberg lettuce 40 green salad 41 corn salad 42 lettuce 43 leaflettuce 44 salad Mizuno 45 Oakleaf lettuce 46 radicchio salad 47 romainelettuce 48 salad Friess 49 salad mix 50 celery 51 Lemon grass (lemongrass) 52 Italian herbs 53 spicy herbs 54 Dill 55 dandelion flowers 56flowers 57 lavender flowers 58 chicory 59 thyme 60 Ramson 61 saffron 62rosehips 63 chives 64 spinach 65 sorrel 66 tarragon 14 Cereals, legumesand flours 1 beans 2 mung beans 3 bulgur 4 puffed rice 5 buckwheat green6 Quinoa 7 buckwheat 8 corn grits 9 semolina 10 Oats 11 pearl barley 12cereal wheat 13 couscous 14 Flour 15 buckwheat flour 16 chestnut flour17 corn flour 18 almond flour 19 Chickpea flour 20 oat flour 21 wheatflour 22 rye flour 23 rice flour 24 Chickpeas 25 Bran 26 Millet 27Figure 28 Figure baya 29 basmati rice 30 brown rice 31 wild rice 32Round grain rice 33 semola (flour made from durum wheat) 34 Beans 35white beans 36 red beans 37 buckwheat flakes 38 cereal grains 39 oatflakes 40 Lentils 41 Barley 15 Spices and Seasonings 1 star anise 2white pepper 3 Vanillin 4 Vanilla 5 vanilla essence 6 vanilla powder 7Wasabi 8 Caltrop 9 garam masala 10 Carnation 11 cloves minced 12 Mustard13 sweet mustard 14 allspice peas 15 grain mustard 16 Cumin 17 groundginger 18 Capers 19 Cardamom 20 Curry 21 Coriander 22 ground coriander23 Cinnamon 24 coffee essence 25 balsamic cream 26 Sesame 27 Turmeric 28bay leaf 29 lemon pepper 30 poppy seed 31 Olives 32 olives dry 33avocado oil 34 anchovy butter 35 peanut oil 36 mustard oil 37 oil forfrying 38 scented oil 39 grapeseed oil 40 canola oil 41 corn oil 42sesame oil 43 linseed oil 44 olive oil 45 Peanut butter 46 sunflower oil47 lean oil 48 vegetable oil 49 oil, refined 50 oil seed-bearing 51soybean oil 52 truffle oil 53 oil pumpkin 54 almonds hammers 55 misopaste 56 sea salt 57 Nutmeg 58 Olives 59 Ligurian olives 60 hot redpepper 61 hot peppers 62 Fenugreek 63 Paprika 64 lemongrass paste 65peperoncini 66 pepper pink polka dots 67 Chili 68 Dried chili peppers 69mustard powder 70 seasoning fish 71 baking powder 72 rosemary 73 pinkground pepper 74 Sugar 75 vanilla sugar 76 brown sugar 77 sugarmuskovado 78 sugar cane 79 powdered sugar 80 nasturtium seeds 81 Nigellaseeds 82 fennel seeds 83 spice mix “taco” 84 Soda 85 ginger juicesqueezed 86 lemon juice 87 Salt 88 citrate 89 grape sauce 90 saucenarsharab 91 ponzu sauce 92 soy sauce 93 tomato sauce 94 chili sauce 95Spices 96 sumac 97 thyme 98 cumin 99 Mediterranean herbs 100 Frenchherbs 101 vinegar 102 balsamic vinegar 103 wine vinegar 104 white winevinegar 105 red wine vinegar 106 cherry vinegar 107 raspberry vinegar108 rice vinegar 109 apple cider vinegar 110 hops suneli 111 Savory 112chutney 113 black pepper 114 black pepper peas 115 dry garlic 116 Sage16 PREPARED PRODUCTS 1 canned pineapple 2 canned artichokes 3 Marinatedartichokes 4 baguette 5 Loaf 6 Bars of chocolate 7 meringue 8 biscuit 9beans, canned 10 Bun 11 buns for hamburgers 12 Broth 13 beef broth 14chicken broth 15 fish broth 16 Jam 17 Apricot jam 18 lingonberry jam 19cherry jam 20 black currant jam 21 raspberry jam 22 blueberry jam 23Wafer 24 canned cherry 25 Glaze 26 Dijon mustard 27 croutons 28marinated mushrooms 29 Demiglas apple 30 Yeast 31 Jelly 32 leaven 33marshmallows 34 crushed tomatoes in juice 35 pickled ginger 36 Cocoa 37marinated cactus 38 Pickled capers 39 sour cabbage 40 sea kale 41 Kimchi42 wafer cakes 43 gherkins 44 natural coffee 45 instant coffee 46Crackers 47 Chocolate Crumb 48 Croissant 49 bouillon cubes 50 cannedcorn 51 marinated corn 52 Pita 53 Lanspik 54 Ice 55 Letcho 56 lasagnasheets 57 canned salmon 58 pickled onions 59 canned mandarins 60marshmallow 61 hazelnut oil 62 sweet curd 63 Yoghurt 64 Honey 65 honeyin the comb 66 Mix ginger 67 condensed milk 68 condensed milk boiled 69milk powder 70 pickled carrots 71 ice cream 72 vanilla ice cream 73chocolate ice cream 74 salted cucumber 75 pickled cucumbers 76 pickledcucumbers 77 Pecans 78 beet broth 79 corn sticks 80 bread sticks 81tomato paste 82 Pasta Chocolate 83 Pate 84 frozen dumplings 85 hotpepper pickled 86 canned peaches 87 Cookies 88 Biscuit 89 CookiesSavoiardi 90 chocolate cookies 91 Pita 92 Supplements 93 tomatoes injuice 94 canned tomatoes 95 Popcorn 96 Prosciutto 97 Gingerbread 98mango puree 99 mashed potatoes 100 tomato puree 101 apple puree 102pickle cucumber 103 Roll 104 Pickled beets 105 pork jerky 106 sugarsyrup 107 whipped cream 108 cream of coconut 109 Malt 110 Sorbet 111barbecue sauce 112 sauce bearnez 113 Béchamel 114 Worcestershire sauce115 sauce Demiglas 116 sauce for soups “Bright udon” 117 sweet and soursauce 118 Salsa 119 sweet sauce 120 chocolate sauce 121 berry sauce 122asparagus, soya 123 caramel chips 124 crushed crackers 125 Tartlets 126Tahini 127 pasta for lasagna 128 dough for ravioli 129 pizza dough 130yeast dough 131 dough kataifi 132 shortbread dough 133 pastry dough 134puff pastry 135 dough dry 136 filo pastry 137 dried tomatoes 138Tortilla 139 Toast 140 Tofu 141 tuna fish oil 142 tuna canned in its ownjuice 143 Tahini 144 Rice Stuffing 145 Canned beans 146 white bread 147toast bread 148 rye bread 149 sweet bread 150 black bread 151 rye bread152 corn flakes 153 ciabatta 154 tea Away 155 potato chips 156 cornchips 157 Marinated mushrooms 158 chocolate corn balls 159 Chocolate 160white chocolate 161 bitter chocolate 162 milk chocolate 163 darkchocolate

TABLE C Lists of Food Preparation Methods and Equipment, Cuisine andBases A list of food preparation methods 1; “0”; “The fried” 2; “0”; Theboiled” 3; “0”; The stewed” 4; “0”; “The baked” 5; “0”; “The cut” A listof Equipment 1; “0”; “ KITCHEN ACCESSORIES” 2; “1”; “funnels” 3; “2”;“stainless steel funnel” 4; “2”; “plastic funnel” 5; “2”; “siliconefunnel” 6; “2”; “convertible funnel” 7; “1”; “colanders” 8; “7”;“quadratic colanders” 9; “7”; “oval ladle-vases” 10; “7”; “colanderswith folding handles” 11; “7”; “flat colander” 12; “7”; “plasticcolanders” 13; “7”; “small round colanders” 14; “7”; “suspendedcolanders” 15; “7”; “cover-colander” 16; “7”; “stainless steel andaluminum colanders” 17; “7”; “cone colanders” 18; “1”; “KitchenAppliances” 19; “18”; “whisk” 20; “18”; “scoop spatula” 21; “20”; “cookspatula” 22; “20”; “spatula with slots” 23; “20”; “confectioneryspatula” 24; “18”; “spoons” 25; “24”serving spoon” 26; “24”;“spoon-tongs” 27; “24”; “spoon with slots” 28; “24”; “spoon for rice”29; “24”; “ladle spoon” 30; “24”; “ice cream spoon” 31; “24”; “honeyspoon” 32; “24”; “spaghetti spoon” 33; “24”; “serving spoon” 34; “18”;“confectionery syringe for cookies and cream” 35; “18”; “soup ladle” 36;“18”; “potato masher” 37; “18”; “skimmer” 38; “18”; “Meat fork” 39;“18”; “brush” 40; “18”; “coffee filter” 41; “18”; “whisk” 42; “18”;“silicone brush” 43; “18”; “silicone juicer” 44; “18”; “earthen saucer”45; “18”; “tea filter” 46; “18”; “pump dispenser for oil and vinegar”47; “18”; “clip for silicone spoon for the edge of the pan” 48; “18”;“transformed spoons for salad” 49; “18”; “device for cherry seedsremoving” 50; “18”; “sink mat” 51; “18”; “ties for rolls” 52; “18”;“dough mini-scraper” 53; “18”; “grill tongs” 54; “18”; “spaghetti tongs”55; “18”; “ice tongs” 56; “18”; “sugar tongs” 57; “18”; “package clip”58; “18”; “package clip” 59; “18”; “citrus spray” 60; “18”; “Doughpress” 61; “18”; “scoop for bulk” 62; “18”; “salad serving tongs(tweezers)” 63; “18”; “accessories for tubes” 64; “18”; “pestle” 65;“18”; “mortar” 66; “18”; “roller for cutting of the rings” 67; “18”;“opener for caps” 68; “18”; “meat tenderizer; meat softener” 69; “18”;“egg yolk separator” 70; “18”; “apron” 71; “18”; “tools for decoration”72; “18”; “jar for oil and vinegar” 73; “18”; “mug for milk boiling” 74;“18”; “napkins” 75; “18”; “tablecloth” 76; “18”; “marker for glasses”78; “18”; “basket” 79; “18”; “meat tenderizer” 80; “18”; “cocotte” 81;“18”; “brush for washing of the vegetables” 82; “18”; “lids for cups”83; “18”; “rope for baking” 84; “18”; “jar for herbs storage” 86; “18”;“scraper for glass ceramic plates” 87; “18”; “Teapot for tea” 88; “18”;“clothespin for notes on the fridge” 89; “18”; “railing systems” 90;“18”; “hanger for kitchen tools” 91; “18”; plunger with not adheringsurface” 92; “18”; “silicone plunger” 93; “18”; “rolling pin withadjustable thickness” 94; “18”; “vacuum bags with pump” 95; “18”; “gaslighter” 96; “18”; “bone forceps” 97; “1”; “kitchen timers thermometers”98; “97”; “timer for meat roasting” 99; “97”; “digital thermometer” 100;“97”; “holder for thermometer” 101; “97”; “meat thermometer” 102; “97”;“digital timer” 103; “97”; “electr. digital timer” 104; “97”; “aramelthermometer” 105; “1”; “Mills for spices” 106; “105”; “mill for blackpepper” 107; “105”; “electric mill” 108; “105”; “combined mill forpepper and salt (2 in 1)” 109; “105”; “mill for spices” 110; “105”;“mill for greens” 111; “1”; “Measuring utensils” 112; “111”; “Measuringcontainer (plastic bottle)” 113; “111”; “measuring jar” 114; “111”;“measuring jug” 115; “111”; “measuring bowl” 116; “111”; “mechanicaldispenser for ice cream” 117; “1”; “mechanical mixers” 118; “1”; “bowl”119; “118”; “metal bowl” 120; “118”; “stainless steel bowl” 121; “118”;“plastic bowl” 122; “118”; “plastic bowl” 123; “118”; “bowls for food”124; “1”; “sets” 125; “124”; “wine set” 126; “124”; “sets for spices”127; “124”; “cupcakes baking set” 128; “124”; “accessory kit for baking”129; “124”; “set of bar tools” 130; “124”; “set of kitchen tools” 131;“124”; “Set for eggs and pancakes baking” 132; “1”; “Slicing and cuttingof products” 133; “132”; “cutter” 134; “132”; “holder for onionscutting” 135; “132”; “cutting boards” 136; “132”; “universalprofessional knives” 137; “132”; “kitchen shears” 138; “132”; “hatchet”139; “132”; “meat hatchet” 140; “132”; “Hammer for meat with hatchet”141; “132”; “hoe” 142; “132”; “Hammer for meat” 143; “132”; “knives”144; “143”; “knife for greens” 145; “143”; “knife for oranges” 146;“143”; “knife for kiwi” 147; “143”; “knife for pineapple” 148; “143”;“Spiral knife for carrots” 149; “143”; “multifunctional knife” 150;“143”; “vegetable knife” 151; “143”; “Pizza Cutter” 152; “143”;“universal knife” 153; “143”; “knife for slicing” 154; “143”; “cookknife” 155; “143”; “gastronomic knife” 156; “143”; “opener” 157; “143”;“Cheese knife” 158; “143”; “boning knife” 159; “143”; “lettuce knife”160; “143”; “knife for steaks” 161; “143”; “butcher knife” 162; “143”;“shredding knife” 163; “143”; “bread knife” 164; “143”; “fish knife”165; “143”; “knife for sandwiches” 166; “143”; “Santoku knife” 167;“143”; “knife for fruit coring” 168; “143”; “Butter knife” 169; “169”;“openers” 170; “169”; “tin-opener” 171; “169”; “corkscrew” 172; “169”;“corkscrew on a stand” 173; “169”; “lever corkscrew” 174; “169”;“folding corkscrew” 175; “169”; “opener for waiter” 178; “494”; “standsfor hot” 179; “494”; “stand for kitchen utensils storing” 180; “494”;“toothpick holder” 181; “494”; “Bottle holder” 182; “494”; “Holder forcapsules” 183; “494”; “stand for spoon” 184; “494”; “stand for coffeecapsules” 185; “494”; “Coasters” 186; “494”; “Napkin holder” 187; “494”;“stand for eggs” 188; “494”; “stand for openers” 189; “494”; “stand forscoops” 190; “494”; “stand for cooking and serving of eggs” 191; “494”;“stand for ladle” 192; “494”; “Holder for paper towels” 193; “494”;“Transforming stand for kitchen appliances” 194; “494”; “stand for mug”195; “494”; “stand mugs and saucers” 196; “494”; “stand for kitchenknives” 197; “494”; “stand for chicken” 198; “494”; “napkin-stand” 199;“494”; “heated stand” 200; “494”; “stands for cake” 201; “1”;“Appliances for peeling and cutting” 202; “201”; “grater for vegetables”203; “305”; “grater” 204; “201”; “garlic masher” 205; “201”; “eggcutter” 206; “201”; “Manual vegetable cutter” 207; “201”; “Peeler forvegetables” 208; “201”; “Nutcracker” 209; “201”; “The device forseparating the yolks from the whites” 210; “201”; “grasping for carrotscleaning” 211; “201”; “scraper fish scales” 212; “201”; “cutter forfruits” 213; “201”; “oller for holes” 214; “201”; “tongs for fish bones”215; “201”; “spiral vegetable cutter” 216; “1”; “Bottle Caps” 217;“216”; “champagne cork (stopper)” 218; “216”; “stoppers for wine” 219;“216”; “The opener to remove the crown corks from bottles” 220; “1”;“sieves” 221; “220”; “sieve for tea” 222; “220”; “sieve-tongs for tea”223; “220”; “Strainer for spices” 224; “220”; “Strainer for tea” 225;“220”; “Universal sieve” 226; “220”; “flour sieve” 228; “220”; “TheChinese sieve with a mesh insert” 229; “220”; “sieve with support” 230;“220”; “Mug-sieve for flour” 231; “220”; “sieve on the handle” 232; “1”;“Salt and pepper shakers” 233; “282”; “container for seasoning” 234;“232”; “salt cellar” 235; “232”; “containers for oil and vinegar” 236;“1”; “Dish dryers” 237; “236”; “salad dryer” 238; “236”;“dryer-placemat” 239; “236”; “dryer for crockery and cutlery” 240; “1”;“Cutlery Accessories” 241; “240”; “cutlery tray” 242; “240”; “cutleryholder” 243; “240”; “cutlery container” 244; “240”; “strainer forcutlery” 245; “240”; “wall hanger for kitchen tools” 246; “240”;“cutlery organizer” 247; “240”; “mat for cutlery” 248; “240”; “slidingtray for cutlery” 249; “240”; “dryer for cutlery” 250; “240”; “glass forcutlery” 251; “240”; “napkin for the cutlery” 252; “240”; “case forcutlery” 253; “240”; “tray for cutlery” 254; “240”; “mitten-potholder”255; “240”; “box for cutlery” 256; “240”; “full-size rack (cassette) forcutlery” 257; “240”; “Stand without containers for cutlery” 258; “240”;“cassette for cutlery” 259; “240”; “container for cutlery” 260; “240”;“station for cutlery” 261; “240”; “Shelf for cutlery” 262; “1”;“Decorations for cocktails” 263; “262”; “ducts” 264; “262”; “sticks”266; “496”; “molds for ice” 267; “496”; “molds for children” 268; “496”;“Molds for shaping products” 269; “496”; “Molds for dumplings” 271;“497”; “Measuring container” 272; “497”; “A mixing container with thedispenser” 273; “497”; “Measuring container with the funnel” 274; “497”;“beaker” 275; “497”; “scoop” 276; “1”; “kitchen scissors” 277; “276”;“Scissors for BBQ” 278; “276”; “Kitchen scissors with bottle opener”279; “276”; “Scissors for greens” 280; “276”; “Kitchen multipurposescissors” 281; “276”; “Kitchen scissors for poultry” 282; “1”; “utensilfor storage” 283; “282”; “container for storage” 284; “282”; “ Bottlesfor liquid spices oils” 285; “282”; “jars for storage” 286; “282”;“lunchbox” 287; “282”; “foldable lunchbox” 288; “282”; “jar for hermeticstorage of bulk products' 289; “282”; “Sprayer for oil/vinegar” 290;“282”; “jar for bulk products” 291; “282”; “containers for spices” 293;“282”; “Container for tea” 294; “1”; “potholders” 295; “294”;“oven-glove” 296; “294”; “silicone potholders” 297; “294”; “dishcloth”298; “1”; “railing with hooks” 299; “1”; “silicone mats” 300; “299”;“baking mat” 301; “299”; “mat for baking cakes” 302; “299”; “mat fordrying of the glasses” 303; “299”; “cooking mat” 304; “299”; “Mat fordrying of the dishes” 305; “1”; “graters presses rubbing machines” 306;“305”; “grater with a handle” 308; “305”; “multifunction grater” 309;“305”; “grater shredder” 310; “305”; “grind for the green” 311; “305”;“grind for the garlic” 312; “305”; “Slicer for tomatoes” 313; “305”;“grater with rotating drums” 314; “305”; “universal device for grinding”315; “305”; “mechanical grater” 316; “305”; “garlic peeling tube” 317;“305”; “rubbing machine” 318; “305”; “press for vegetables” 319; “305”;“press for garlic” 320; “305”; “press for hamburgers” 321; “1”; “knifesharpener” 322; “321”; “electric sharpener” 323; “321”; “sharpeningstone” 324; “321”; “ceramic sharpener” 325; “1”; “breadbox” 326; “1”;“lattice with legs” 327; “339”; “Flatware” 328; “327”; “for alcohol”329; “540”; “Cognac set with the batcher” 330; “540”; “Glasses souvenir”331; “540”; “Glasses” 332; “540”; “Bucket for ice” 333; “540”; “Shotglasses” 334; “540”; “Bucket for champagne” 335; “540”; “Wine glasses”336; “540”; “decanter” 337; “540”; “tray” 338; “540”; “Support under abottle” 339; “327”; “tableware” 340; “339”; “first course dish” 341;“339”; “dish for bouillon” 342; “339”; “bouillon bowl” 343; “339”;“oiler” 344; “339”; “round dish” 345; “339”; “duck pan” 346; “339”; “Setfor making chocolate fondue 347; “339”; “Set for making cheese fondue”348; “339”; “salad bowl” 349; “339”; “dish for cake” 350; “339”;“compartmental dish” 351; “339”; “set of cutlery” 352; “339”; “servingspoon and fork” 353; “339”; “dish with lid” 354; “339”; “steam table”355; “374”; “ice-cream bowl” 357; “339”; “saucer” 358; “339”; “saucerfor jam” 359; “339”; “mustard-pot” 360; “339”; “pepper-pot” 361; “339”;“ash-pot” 362; “339”; “deep table plate” 363; “339”; “dinner plate” 364;“339”; “snack plate” 365; “339”; “deep dessert plate” 366; “339”;“dessert plate” 367; “339”; “plate for pies” 368; “339”;“horseradish-pot” 369; “327”; “Utensils for table” 370; “369”; “Pad fortableware” 371; “369”; “serving mat” 372; “369”; “serving tray” 373;“369”; “glass burner” 374; “327”; “Dishes for tea coffee desert” 375;“374”; “sugar-bowl” 376; “374”; “mug” 377; “374”; “mug with teapot” 378;“374”; “mug with stand” 379; “374”; “mug with lid” 380; “374”; “tea set”381; “374”; “dish” 382; “374”; “french-press” 383; “374”; “teapot” 384;“374”; “teapot with strainer” 385; “374”; “glass teapot” 387; “374”;“multifunctional vase” 388; “540”; “Glasses” 389; “374”; “soup bowl”390; “374”; “wicker basket” 391; “374”; “vase 3-tier” 393; “374”;“napkin rings” 394; “374”; “pannier for fruits” 395; “374”; “table trashbasket” 396; “374”; “biscuit dish” 397; “374”; “candy dish” 398; “374”;“coffee sets” 399; “327”; “CUTLERY” 437; “0”; “Kitchen appliances” 438;“437”; “aerogrill” 439; “437”; “blenders grinder” 440; “437”; “coffeeMaker” 441; “437”; “coffee grinder (coffee mill)” 442; “437”; “FoodProcessor” 443; “437”; “mixer” 444; “437”; “mini oven” 445; “437”;“multicooker” 446; “437”; “meat grinders” 447; “437”; “steamers” 448;“437”; “Raclette grill” 449; “437”; “Juicers” 450; “437”; “toasters”451; “437”; “egg cooker” 452; “437”; “electric range” 453; “437”;“electric induction stove” 454; “437”; “electric kettle” 455; “437”;“thermopots” 456; “437”; “bread makers” 457; “437”; “microwaves” 458;“437”; “weights for kitchen” 459; “437”; “electric driers” 461; “0”;“Children's dishes” 462; “461”; “Children Sets for baking” 463; “461”;“Children cutlery” 464; “461”; “Children thermoses” 465; “461”;“Children Sets of dishes” 488; “437”; “deep fryer” 491; “339”; “bakingsheet” 494; “1”; “stand and holders” 495; “220”; “sieve to form the“Bird's Nest”” 496; “1”; “mold” 497; “1”; “Measuring container” 498;“339”; “pan” 499; “339”; “frying pan” 500; “437”; “Cookware forinduction cookers” 501; “437”; “Juice cookers” 502; “437”; “Milk cooker”503; “437”; “Covers/splash screens” 504; “437”; “Microwave cookware”505; “437”; “Braziers roasters” 506; “437”; “Turk” 507; “437”; “Dumpling(manti) cookers” 508; “437”; “Sets” 509; “437”; “Samovars” 510; “437”;“Kasans” 511; “437”; “Electric stove” 512; “437”; “Casseroles (pans)”513; “512”; “casseroles (pans) with non-stick coating” 514; “512”;“aluminum casseroles (pans)” 515; “512”; “Stainless steel casseroles(pans)” 516; “512”; “Enameled casseroles (pans)” 517; “512”; “Tefloncoated casseroles (pans)” 518; “512”; “Heat-proof glass casseroles(pans)” 519; “512”; “Ladles” 520; “512”; “Ceramic casseroles (pans)”521; “512”; “Set of casseroles (pans)” 522; “512”; “Pressure cooker”523; “512”; “Pan-steamer” 524; “512”; “casseroles for induction cookers”525; “512”; “Pan-fryer” 526; “512”; “Cast iron casserole (pot)” 527;“512”; “Titanium casserole” 528; “437”; “Frying pans skillet” 529;“528”; “Frying pan with ceramic coating” 530; “528”; “Frying pans withnon-stick coating” 531; “528”; “Frying pan with removable handle” 532;“528”; “Stewpots” 533; “528”; “Frying pans for grill” 534; “528”; “Wok”535; “528”; “Pancake pans” 536; “528”; “Electric frying pans” 537;“528”; “Cast iron skillet” 538; “528”; “Multifunctional frying pan” 539;“528”; “Titanium frying pan” 540; “437”; “Drinkware” 541; “540”; “Wineglasses” 542; “540”; “Water glasses” 543; “540”; “Beer glasses” 544;“540”; “Kegs” 545; “540”; “Carafes” 546; “540”; “Decanters” 547; “540”;“Jugs” 548; “540”; “Shots” 549; “540”; “Wine glasses for champagne” 550;“540”; “Glasses for brandy/cognac” 551; “540”; “Wine glasses for acocktail/martini” A list of Cuisine 1; “0”; “Abkhaz” 2; “0”;“Australian” 3; “0”; “Austrian” 4; “0”; “Azerbaijan” 5; “0”; “Albanian”6; “0”; “Algerian” 7; “0”; “American” 8; “0”; “English” 9; “0”; “Arabic”10; “0”; “Argentine” 11; “0”; “Armenian” 12; “0”; “Bashkir” 13; “0”;“Belarusian” 14; “0”; “Belgian” 15; “0”; “Bulgarian” 16; “0”; “Bosnian”17; “0”; “Brazilian” 18; “0”; “Hungarian” 19; “0”; “Venezuelan” 20; “0”;“Vietnamese” 21; “0”; “Greek” 22; “0”; “Georgian” 23; “0”; “Danish” 24;“0”; “Jewish” 25; “0”; “Israeli” 26; “0”; “Indian” 27; “0”; “Indonesian”28; “0”; “Jordanian” 29; “0”; “Iraqi” 30; “0”; “Iranian” 31; “0”;“Irish” 32; “0”; “Icelandic” 33; “0”; “Spanish” 34; “0”; “Italian” 35;“0”; “Cambodian” 36; “0”; “Canadian” 37; “0”; “Cypriot” 38; “0”;“Chinese” 39; “0”; “Colombian” 40; “0”; “Korean” 41; “0”; “Creole” 42;“0”; “Costa Rica” 43; “0”; “Latvian” 44; “0”; “Lebanese” 45; “0”;“Libyan” 46; “0”; “Lithuanian” 47; “0”; “Macedonian” 48; “0”;“Malaysian” 49; “0”; “Moroccan” 50; “0”; “Mexican” 51; “0”; “Moldavian”52; “0”; “Mongolian” 53; “0”; “German” 54; “0”; “Dutch” 55; “0”; “NewZealand” 56; “0”; “Norwegian” 57; “0”; “Ossetian” 58; “0”; “Pakistani”59; “0”; “Palestinian” 60; “0”; “Panamanian” 61; “0”; “Peruvian” 62;“0”; “Polish” 63; “0”; “Portuguese” 64; “0”; “Romanian” 65; “0”;“Russian” 66; “0”; “Serbian” 67; “0”; “Singaporean” 68; “0”; “Syrian”69; “0”; “Slovak” 70; “0”; “Slovenian” 71; “0”; “Thai” 72; “0”; “Tatar”73; “0”; “Tibetan” 74; “0”; “Tunisian” 75; “0”; “Turkish” 76; “0”;“Turkmen” 77; “0”; “Ukrainian” 78; “0”; “Philippine” 79; “0”; “Finnish”80; “0”; “French” 81; “0”; “Croatian” 82; “0”; “Montenegrin” 83; “0”;“Czech” 84; “0”; “Chilean” 85; “0”; “Chuvash” 86; “0”; “Chukotka” 87;“0”; “Swedish” 88; “0”; “Swiss” 89; “0”; “Scottish” 90; “0”;“Ecuadorian” 91; “0”; “Estonian” 92; “0”; “Japanese” 93; “0”; “Raw fooddiet” 94; “0”; “European” 95; “0”; “International” 96; “0”;“Multinational” 97; “0”; “Lean” 98; “0”; “Caucasian” 99; “0”; “Children”A list of bases: 1; “0”; “Meat and meat products” 2; “1”; “Basturma” 3;“1”; “Fat” 4; “1”; “brisket cooked and smoked” 5; “1”; “Hare” 6; “1”;“leather duck” 7; “1”; “Sausage” 8; “1”; “Sausages” 9; “1”; “Sausages“Hunting party”” 10; “1”; “Horsemeat” 11; “1”; “Bones with bone marrow”12; “1”; “Roe” 13; “1”; “Rabbit” 14; “1”; “Meat” 15; “1”; “Moosemeat”16; “1”; “Venison” 17; “1”; “Liver” 18; “1”; “Kidney” 19; “1”; “Smokedribs” 20; “1”; “Salami” 21; “1”; “Sausages” 22; “1”; “Cervelat” 23; “1”;“Sausages” 24; “1”; “Hungarian smoked bacon” 25; “l”; “bacon fat-tailed”26; “1”; “Steak” 27; “1”; “ribeye steak” 28; “1”; “Farce” 29; “1”;“crocodile fillet” 30; “1”; “Jamon” 31; “1”; “Choriso (spanish sausage)”32; “1”; “Skewers” 33; “1”; “Sowbelly” 34; “1”; “Deer tongue” 35; “1”;“LAMB” 36; “35”; “breast of lamb” 37; “35”; “loin of lamb” 38; “35”;“blade lamb” 39; “35”; “veal brains” 40; “35”; “mutton ham” 41; “35”;“veal ham” 42; “35”; “leg of lamb” 43; “35”; “Heel muscle mutton” 44;“35”; “lamb offal” 45; “35”; “veal kidneys” 46; “35”; “lamb chops” 47;“35”; “gras cow” 48; “35”; “veal heart” 49; “35”; “lamb testicles” 50;“35”; “VEAL” 51; “35”; “veal cheeks” 52; “35”; “minced lamb” 53; “35”;“minced veal” 54; “35”; “veal tail” 55; “35”; “veal tongue” 56; “35”;“eggs bullish” 57; “1”; “BEEF” 58; “57”; “beef brisket” 59; “57”; “” 60;“57”; “beef (sirloin)” 61; “57”; “beef on the bone” 62; “57”; “beef eyemuscle” 63; “57”; “legs of beef” 64; “57”; “ham beef” 65; “57”; “grasbeef” 66; “57”; “beef ribs” 67; “57”; “beef heart” 68; “57”; “mincedbeef” 69; “57”; “tail beef” 70; “57”; “beef tongue” 71; “1”; “PORK” 72;“71”; “bacon” 73; “71”; “smoked bacon” 74; “71”; “pork” 75; “71”; “ham”76; “71”; “pork brisket” 77; “71”; “smoked pork belly” 78; “71”; “PorkIntestine” 79; “71”; “legs of pork” 80; “71”; “boar ham” 81; “71”; “porkham” 82; “71”; “pork ribs” 83; “71”; “knuckle of pork” 84; “71”; “fat”85; “71”; “pork (pork neck or loin)” 86; “71”; “pork ears” 87; “71”;“minced pork” 88; “71”; “pig tail” 89; “71”; “pork tongue” 90; “0”;“Birds” 91; “90”; “garshnep” 92; “90”; “turkey breast” 93; “90”;“chicken breast” 94; “90”; “chicken breast smoked” 95; “90”; “duckbreast” 96; “90”; “Goose” 97; “90”; “chicken ventricles” 98; “90”;“turkey” 99; “90”; “turkey wings” 100; “90”; “chicken wings” 101; “90”;“chicken” 102; “90”; “smoked chicken” 103; “90”; “grouse” 104; “90”;“coot” 105; “90”; “duck leg” 106; “90”; “crow's feet” 107; “90”;“chicken legs” 108; “90”; “chicken ham” 109; “90”; “quail” 110; “90”;“gras chicken” 111; “90”; “chicken giblets” 112; “90”; “grouse” 113;“90”; “chicken hearts” 114; “90”; “duck” 115; “90”; “smoked duck” 116;“90”; “Pheasant” 117; “90”; “minced chicken” 118; “90”; “chicken fillet”119; “90”; “foie gras” 120; “90”; “chicken” 121; “90”; “chicken gutted”122; “90”; “neck duck” 123; “0”; “FISH and SEAFOOD” 124; “123”;“anchovies” 125; “123”; “arctic char” 126; “123”; “mullet” 127; “123”;“Black Sea goby” 128; “123”; “shrimp head” 129; “123”; “Butterfish” 130;“123”; “scallops” 131; “123”; “dorado” 132; “123”; “ruff” 133; “123”;“caviar” 134; “123”; “red caviar” 135; “123”; “Tobiko caviar” 136;“123”; “squid” 137; “123”; “flounder” 138; “123”; “cuttlefish” 139;“123”; “carp” 140; “123”; “sprat” 141; “123”; “smelt” 142; “123”; “crabsticks” 143; “123”; “Shrimps” 144; “123”; “King shrimps” 145; “123”;“Salad shrimps” 146; “123”; “Tiger prawn” 147; “123”; “Bream” 148;“123”; “salmon” 149; “123”; “Smoked salmon” 150; “123”; “Mussels” 151;“123”; “Mussels with shells' 152; “123”; “Pollock” 153; “123”;“Molluscs” 154; “123”; “Sea food” 155; “123”; “Sea fish” 156; “123”;“sole (fish)” 157; “123”; “Crab meat” 158; “123”; “Krill meat” 159;“123”; “Burbot” 160; “123”; “Frog legs” 161; “123”; “Perch” 162; “123”;“Lobster” 163; “123”; “cisco” 164; “123”; “sturgeon” 165; “123”;“octopus” 166; “123”; “baby octopus” 167; “123”; “shrimp broth” 168;“123”; “halibut” 169; “123”; “Pangasius” 170; “123”; “cod liver oil”171; “123”; “haddock” 172; “123”; “crayfish” 173; “123”; “driedcrustaceans” 174; “123”; “Hot smoked fish” 175; “123”; “red fish salted”176; “123”; “swordfish” 177; “123”; “saury” 178; “123”; “sardines” 179;“123”; “herring” 180; “123”; “salmon” 181; “123”; “smoked salmon” 182;“123”; “salted salmon” 183; “123”; “seabass” 184; “123”; “whitefish”185; “123”; “ramp” 186; “123”; “mackerel” 187; “123”; “smoked mackerel”188; “123”; “sheatfish” 189; “123”; “starlet” 190; “123”; “walleye” 191;“123”; “Dried seaweed” 192; “123”; “tilapia” 193; “123”; “carp” 194;“123”; “cod” 195; “123”; “Hot smoked cod” 196; “123”; “black cod” 197;“123”; “tuna” 198; “123”; “turbot” 199; “123”; “eel” 200; “123”; “smokedeel” 201; “123”; “snails” 202; “123”; “oysters” 203; “123”; “white fishfillets” 204; “123”; “catfish fillets” 205; “123”; “fillet of carp” 206;“123”; “fish fillet” 207; “123”; “salmon fillet” 208; “123”; “saltedherring fillets” 209; “123”; “perch fillet” 210; “123”; “trout” 211;“123”; “smoked trout” 212; “123”; “Squid Ink” 213; “123”; “cervicalshrimp” 214; “123”; “cervical cancers” 215; “123”; “sprats” 216; “123”;“pike” 217; “0”; “VEGETABLES” 218; “217”; “watermelon” 219; “217”;“Artichokes” 220; “217”; “eggplant” 221; “217”; “yam” 222; “217”;“broccoli tops” 223; “217”; “beet tops” 224; “217”; “broccoli” 225;“217”; “rutabaga” 226; “217”; “galangal” 227; “217”; “peas” 228; “217”;“pea sprouts” 229; “217”; “pea pods” 230; “217”; “green peas” 231;“217”; “daikon” 232; “217”; “melon” 233; “217”; “Ginseng” 234; “217”;“Ginger” 235; “217”; “zucchini” 236; “217”; “feces” 237; “217”;“cabbage” 238; “217”; “Brussels sprouts” 239; “217”; “sauerkraut” 240;“217”; “Chinese cabbage” 241; “217”; “Cabbage” 242; “217”; “Romanescocabbage” 243; “217”; “savoy cabbage” 244; “217”; “cauliflower” 245;“217”; “potatoes” 246; “217”; “young potatoes' 247; “217”; “kohlrabi”248; “217”; “root anise” 249; “217”; “salsify root” 250; “217”; “parsleyroot” 251; “217”; “celery root” 252; “217”; “fresh corn” 253; “217”;“white onion” 254; “217”; “pearl bow” 255; “217”; “onion” 256; “217”;“red onion” 257; “217”; “dry onion” 258; “217”; “small onion” 259;“217”; “Shallots” 260; “217”; “cassava” 261; “217”; “mini corn” 262;“217”; “mini peppers” 263; “217”; “mini-tomatoes” 264; “217”; “carrots”265; “217”; “cucumber” 266; “217”; “parsnips” 267; “217”; “squash” 268;“217”; “bell peppers” 269; “217”; “cayenne pepper” 270; “217”; “freshchili pepper” 271; “217”; “jalapeno peppers” 272; “217”; “tomato” 273;“217”; “pickled tomatoes” 274; “217”; “cherry tomatoes” 275; “217”;“sunflower sprouts” 276; “217”; “wheat germ” 277; “217”; “soybeanseedlings” 278; “217”; “germinated soybeans” 279; “217”; “rhubarb” 280;“217”; “Radish” 281; “217”; “wild radish” 282; “217”; “Turnip” 283;“217”; “beansprouts” 284; “217”; “beet” 285; “217”; “Asparagus” 286;“217”; “chopped tomatoes” 287; “217”; “Sweet” 288; “217”; “Pumpkin” 289;“217”; “green beans” 290; “217”; “Fennel” 291; “217”; “physalis” 292;“217”; “horseradish” 293; “217”; “zucchini” 294; “217”; “garlic” 295;“217”; “endive” 296; “0”; “FRUITS” 297; “296”; “Apricot” 298; “296”;“Avocado” 299; “296”; “quince” 300; “296”; “fresh pineapple” 301; “296”;“Orange” 302; “296”; “banana” 303; “296”; “Hawthorn” 304; “296”;“cranberries” 305; “296”; “grapes” 306; “296”; “Cherry” 307; “296”;“Dried cherries” 308; “296”; “blueberries” 309; “296”; “Garnet” 310;“296”; “Grapefruit” 311; “296”; “pear” 312; “296”; “Blackberry” 313;“296”; “strawberries” 314; “296”; “pomegranate seeds” 315; “296”;“carambola” 316; “296”; “kiwi” 317; “296”; “Strawberry” 318; “296”;“Cranberry” 319; “296”; “coconut” 320; “296”; “gooseberry” 321; “296”;“kumquat” 322; “296”; “Lime” 323; “296”; “lemon” 324; “296”; “Litchi”325; “296”; “raspberries” 326; “296”; “mango” 327; “296”; “Mandarin”328; “296”; “Passionfruit” 329; “296”; “mini pineapple 330; “296”;“Nectarine” 331; “296”; “buckthorn” 332; “296”; “papaya” 333; “296”;“Peach” 334; “296”; “Pomelo” 335; “296”; “Rowan” 336; “296”; “drain”337; “296”; “red currants” 338; “296”; “black currant” 339; “296”;“tamarind” 340; “296”; “Feijoa” 341; “296”; “fruit to taste” 342; “296”;“persimmon” 343; “296”; “cherries” 344; “296”; “Cherry” 345; “296”;“blueberries” 346; “296”; “apple” 347; “296”; “frozen berries” 348;“296”; “juniper berries” 349; “296”; “fresh berries” 350; “0”; “GROCERY”351; “350”; “agar” 352; “350”; “adjika” 353; “350”; “rice paper” 354;“350”; “vanilla extract” 355; “350”; “vermicelli rice” 356; “350”; “eggnoodles” 357; “350”; “algae” 358; “350”; “glucose” 359; “350”; “jam”360; “350”; “raspberry jam” 361; “350”; “fresh yeast” 362; “350”;“gelatin” 363; “350”; “liquid Smokehouse” 364; “350”; “sweetener” 365;“350”; “corn muffins” 366; “350”; “ketchup” 367; “350”; “citric acid”368; “350”; “candy” 369; “350”; “confiture” 370; “350”; “strawberry jam”371; “350”; “food dye” 372; “350”; “starch” 373; “350”; “potato starch”374; “350”; “corn starch” 375; “350”; “bread crumbs” 376; “350”;“Noodles” 377; “350”; “buckwheat noodles” 378; “350”; “Pad Thai noodles”379; “350”; “rice noodles” 380; “350”; “glass noodles” 381; “350”;“noodles harusame” 382; “350”; “egg noodles” 383; “350”; “mayonnaise”384; “350”; “poppy sweet” 385; “350”; “pasta” 386; “350”; “cannellonipasta” 387; “350”; “pasta lumakoni” 388; “350”; “pasta feathers” 389;“350”; “fusilli pasta” 390; “350”; “pumpkin marmalade” 391; “350”;“jujube fruit” 392; “350”; “marzipan” 393; “350”; “mirin” 394; “350”;“coconut milk” 395; “350”; “almond milk” 396; “350”; “soy milk” 397;“350”; “muesli” 398; “350”; “Pasta” 399; “350”; “peanut paste” 400;“350”; “red curry paste” 401; “350”; “tamarind paste” 402; “350”; “TomYam Paste” 403; “350”; “chili paste” 404; “350”; “molasses” 405; “350”;“pectin” 406; “350”; “Penne” 407; “350”; “jam” 408; “350”; “elderberrysyrup” 409; “350”; “vanilla syrup” 410; “350”; “syrup vishnevny” 411;“350”; “ginger syrup” 412; “350”; “caramel syrup” 413; “350”; “maplesyrup” 414; “350”; “strawberry syrup” 415; “350”; “coffee syrup” 416;“350”; “corn syrup” 417; “350”; “raspberry syrup” 418; “350”; “mangosyrup” 419; “350”; “honey syrup” 420; “350”; “almond syrup” 421; “350”;“walnut syrup” 422; “350”; “blackcurrant syrup” 423; “350”; “chocolatesyrup” 424; “350”; “cranberry sauce” 425; “350”; “worcestershire sauce”426; “350”; “pomegranate sauce” 427; “350”; “kimchi sauce” 428; “350”;“pesto” 429; “350”; “fish sauce” 430; “350”; “fish sauce nam pla” 431;“350”; “Tabasco sauce” 432; “350”; “teriyaki sauce” 433; “350”; “saucetkemali” 434; “350”; “oyster sauce” 435; “350”; “sweet chili sauce” 436;“350”; “Japanese walnut sauce” 437; “350”; “spaghetti” 438; “350”; “;“crumbs of white bread” 439; “350”; “breadcrumbs” 440; “350”; “pastrydecorations” 441; “350”; “candied” 442; “0”; “MILK PRODUCTS and EGGS”443; “442”; “yogurt” 444; “442”; “natural yoghurt” 445; “442”; “kefir”446; “442”; “margarine” 447; “442”; “butter” 448; “442”; “melted butter”449; “442”; “milk” 450; “442”; “baked milk” 451; “442”; “buttermilk”452; “442”; “curdled” 453; “442”; “cream” 454; “442”; “sour cream” 455;“442”; “whey” 456; “442”; “Thane” 457; “442”; “curd” 458; “442”; “curdbeaded” 459; “442”; “quail eggs” 460; “442”; “egg” 461; “0”; “mushrooms”462; “461”; “oyster mushrooms” 463; “461”; “” 464; “461”; “ceps” 465;“461”; “Enoki mushrooms” 466; “461”; “Chinese dried mushrooms” 467;“461”; “portobello mushrooms” 468; “461”; “dried mushrooms” 469; “461”;“shiitake mushrooms” 470; “461”; “milkmushrooms” 471; “461”;“chanterelles” 472; “461”; “boletus” 473; “461”; “honey fungus” 474;“461”; “saffron milk cap” 475; “461”; “morels” 476; “461”; “truffles”477; “461”; “meadow mushrooms” 478; “0”; “CHEESE” 479; “478”; “cheese”480; “478”; “cheese Adyghe” 481; “478”; “brie cheese” 482; “478”; “fetacheese” 483; “478”; “Burrata cheese” 484; “478”; “Gouda cheese” 485;“478”; “Dutch cheese” 486; “478”; “blue cheese” 487; “478”; “Gorgonzola”488; “478”;” ; “grana padano cheese” 489; “478”; “Gruyere cheese” 490;“478”; “-”; “Dor Blue cheese” 491; “478”; “Camembert” 492; “478”; “goatcheese” 493; “478”; “cheese sausage” 494; “478”; “mascarpone cheese”495; “478”; “Monterey Jack cheese” 496; “478”; “mozzarella cheese” 497;“478”; “soft cheese” 498; “478”; “goat cheese” 499; “478”; “parmesancheese” 500; “478”; “pecorino cheese” 501; “478”; “processed cheese”502; “478”; “cheese Poshehonsky” 503; “478”; “ricotta cheese” 504;“478”; “Roquefort cheese” 505; “478”; “blue cheese” 506; “478”; “creamcheese” 507; “478”; “suluguni” 508; “478”; “cheese curd” 509; “478”;“feta cheese” 510; “478”; “philadelphia cheese” 511; “478”; “cheddarcheese” 512; “478”; “edam cheese” 513; “478”; “Emmentaler cheese” 514;“0”; “NUTS and DRIED FRUITS” 515; “514”; “peanuts” 516; “514”;“barberry” 517; “514”; “walnuts (peeled)” 518; “514”; “raisins” 519;“514”; “figs” 520; “514”; “Chestnut” 521; “514”; “Dried cranberries”522; “514”; “coconut” 523; “514”; “dried apricots” 524; “514”; “Filbert(hazelnut)” 525; “514”; “almonds” 526; “514”; “nuts” 527; “514”; “pinenuts” 528; “514”; “cashew nuts” 529; “514”; “Dried peaches” 530; “514”;“sunflower seeds” 531; “514”; “pumpkin seeds” 532; “514”; “Dried Fruits”533; “514”; “Dates” 534; “514”; “pistachios” 535; “514”; “hazelnuts”536; “514”; “prunes” 537; “0”; “BEVERAGES” 538; “537”; “water” 539;“537”; “water orange” 540; “537”; “mineral water” 541; “537”; “waterpink” 542; “537”; “GABA-tea” 543; “537”; “Hibiscus” 544; “537”; “kvass”545; “537”; “bread kvass” 546; “537”; “-; “Coke” 547; “537”; “Kuding”548; “537”; “lemonade” 549; “537”; “mate” 550; “537”; “juice” 551;“537”; “carbonated drink” 552; “537”; “Bitter Brandy” 553; “537”;“Rooibos” 554; “537”; “pineapple juice” 555; “537”; “orange juice” 556;“537”; “birch juice” 557; “537”; “grape juice” 558; “537”; “cherryjuice” 559; “537”; “pomegranate juice” 560; “537”; “strawberry juice”561; “537”; “cranberry juice” 562; “537”; “gooseberry juice” 563; “537”;“lime juice” 564; “537”; “mango juice” 565; “537”; “tangerine juice”566; “537”; “peach juice” 567; “537”; “currant juice” 568; “537”;“tomato juice” 569; “537”; “apple juice” 570; “537”; “sprite” 571;“537”; “tonic” 572; “537”; “tea white” 573; “537”; “tea yellow” 574;“537”; “green tea” 575; “537”; “red tea” 576; “537”; “Puer tea” 577;“537”; “Puer tea in Mandarin” 578; “537”; “oolong tea” 579; “537”;“black tea” 580; “537”; “Espresso” 581; “0”; “ALCOHOL” 582; “581”;“Balm” 583; “581”; “Bitter” 584; “581”; “brandy” 585; “581”; “bourbon”586; “581”; “vermouth” 587; “581”; “wine” 588; “581”; “white wine” 589;“581”; “sparkling wine” 590; “581”; “red wine” 591; “581”; “dry redwine” 592; “581”; “wine sangria” 593; “581”; “whiskey” 594; “581”;“Vodka” 595; “581”; “anise vodka” 596; “581”; “grappa” 597; “581”; “gin”598; “581”; “-”; “Irish cream liqueur” 599; “581”; “Calvados” 600;“581”; “Cachaca” 601; “581”; “brandy” 602; “581”; “liqueur” 603; “581”;“orange liqueur” 604; “581”; “coffee liqueur” 605; “581”; “chocolateliqueur” 606; “581”; “Madeira” 607; “581”; “Marsala” 608; “581”;“Martini” 609; “581”; “beer” 610; “581”; “cherry beer” 611; “581”;“port” 612; “581”; “rum” 613; “581”; “white rum” 614; “581”; “black rum”615; “581”; “Sake” 616; “581”; “sambuca” 617; “581”; “cider” 618; “581”;“tequila” 619; “581”; “sherry” 620; “581”; “( )”; “Champagne (Brut)”621; “581”; “schnapps” 622; “0”; “GREENS AND HERBS” 623; “622”; “basil”624; “622”; “basil red” 625; “622”; “bouquet garni” 626; “622”;“oregano” 627; “622”; “greens” 628; “622”; “dried herbs” 629; “622”; “-;“cabbage pak choi” 630; “622”; “chervil” 631; “622”; “cilantro” 632;“622”; “oxalis” 633; “622”; “oat root” 634; “622”; “fresh coriander”635; “622”; “nettle” 636; “622”; “Watercress” 637; “622”; “watercress”638; “622”; “rose petals” 639; “622”; “lemongrass” 640; “622”; “bambooleaves” 641; “622”; “banana leaves” 642; “622”; “grape leaves” 643;“622”; “Grape leaves (salty)” 644; “622”; “kaffir lime leaves” 645;“622”; “lime leaves” 646; “622”; “dandelion leaves” 647; “622”; “greenonion” 648; “622”; “-; “Leek” 649; “622”; “marjoram” 650; “622”; “chard”651; “622”; “melissa” 652; “622”; “lemon balm” 653; “622”; “Mint” 654;“622”; “oregano” 655; “622”; “parsley” 656; “622”; “dried parsley” 657;“622”; “plantain” 658; “622”; “wormwood” 659; “622”; “chopped camomile”660; “622”; “arugula” 661; “622”; “iceberg lettuce” 662; “622”; “greensalad” 663; “622”; “corn salad” 664; “622”; “lettuce” 665; “622”; “leaflettuce” 666; “622”; “salad Mizuno” 667; “622”; “: “Oakleaf lettuce”668; “622”; “radicchio salad” 669; “622”; “romaine lettuce” 670; “622”;“salad Friess” 671; “622”; “salad mix” 672; “622”; “celery” 673; “622”;“Lemon grass (lemon grass)” 674; “622”; “Italian herbs” 675; “622”;“spicy herbs” 676; “622”; “dill” 677; “622”; “dandelion flowers” 678;“622”; “flowers” 679; “622”; “lavender flowers” 680; “622”; “chicory”681; “622”; “thyme” 682; “622”; “Ramson” 683; “622”; “saffron” 684;“622”; “rosehips” 685; “622”; “chives” 686; “622”; “spinach” 687; “622”;“sorrel” 688; “622”; “tarragon” 689; “0”; “Cereals legumes and flours”690; “689”; “beans” 691; “689”; “mung beans” 692; “689”; “bulgur” 693;“689”; “puffed rice” 694; “689”; “buckwheat green” 695; “689”; “Quinoa”696; “689”; “buckwheat” 697; “689”; “corn grits” 698; “689”; “semolina”699; “689”; “oats” 700; “689”; “pearl barley” 701; “689”; “cereal wheat”702; “689”; “couscous” 703; “689”; “flour” 704; “689”; “buckwheat flour”705; “689”; “chestnut flour” 706; “689”; “corn flour” 707; “689”;“almond flour” 708; “689”; “Chickpea flour” 709; “689”; “oat flour” 710;“689”; “wheat flour” 711; “689”; “rye flour” 712; “689”; “rice flour”713; “689”; “chickpeas” 714; “689”; “bran” 715; “689”; “millet” 716;“689”; “Figure” 717; “689”; “Figure baya” 718; “689”; “basmati rice”719; “689”; “brown rice” 720; “689”; “wild rice” 721; “689”; “Roundgrain rice” 722; “689”; “semola (flour made from durum wheat)” 723;“689”; “Beans” 724; “689”; “white beans” 725; “689”; “red beans” 726;“689”; “buckwheat flakes” 727; “689”; “cereal grains” 728; “689”; “oatflakes” 729; “689”; “lentils” 730; “689”; “barley” 731; “0”; “Spices andSeasonings” 732; “731”; “star anise” 733; “731”; “white pepper” 734;“731”; “vanillin” 735; “731”; “vanilla” 736; “731”; “vanilla essence”737; “731”; “vanilla powder” 738; “731”; “wasabi” 739; “731”; “caltrop”740; “731”; “garam masala” 741; “731”; “Carnation” 742; “731”; “clovesminced” 743; “731”; “mustard” 744; “731”; “sweet mustard” 745; “731”;“allspice peas” 746; “731”; “grain mustard” 747; “731”; “Cumin” 748;“731”; “ground ginger” 749; “731”; “capers” 750; “731”; “cardamom” 751;“731”; “curry” 752; “731”; “coriander” 753; “731”; “ground coriander”754; “731”; “cinnamon” 755; “731”; “coffee essence” 756; “731”;“balsamic cream” 757; “731”; “sesame” 758; “731”; “turmeric” 759; “731”;“bay leaf” 760; “731”; “lemon pepper” 761; “731”; “poppy seed” 762;“731”; “olives” 763; “731”; “olives dry” 764; “731”; “avocado oil” 765;“731”; “anchovy butter” 766; “731”; “peanut oil” 767; “731”; “mustardoil” 768; “731”; “oil for frying” 769; “731”; “scented oil” 770; “731”;“grapeseed oil” 771; “731”; “canola oil” 772; “731”; “corn oil” 773;“731”; “sesame oil” 774; “731”; “linseed oil” 775; “731”; “olive oil”776; “731”; “Peanut butter” 777; “731”; “sunflower oil” 778; “731”;“lean oil” 779; “731”; “vegetable oil” 780; “731”; “oil refined” 781;“731”; “oil seed-bearing” 782; “731”; “soybean oil” 783; “731”; “truffleoil” 784; “731”; “oil pumpkin” 785; “731”; “almonds hammers” 786; “731”;“miso paste” 787; “731”; “sea ??salt” 788; “731”; “nutmeg” 789; “731”;“olives” 790; “731”; “Ligurian olives” 791; “731”; “hot red pepper” 792;“731”; “hot peppers” 793; “731”; “fenugreek” 794; “731”; “paprika” 795;“731”; “lemongrass paste” 796; “731”; “peperoncini” 797; “731”; “pepperpink polka dots” 798; “731”; “chili” 799; “731”; “Dried chili peppers”800; “731”; “mustard powder” 801; “731”; “seasoning fish” 802; “731”;“baking powder” 803; “731”; “rosemary” 804; “731”; “pink ground pepper”805; “731”; “sugar” 806; “731”; “vanilla sugar” 807; “731”; “brownsugar” 808; “731”; “sugar muskovado” 809; “731”; “sugar cane” 810;“731”; “powdered sugar” 811; “731”; “nasturtium seeds” 812; “731”;“Nigella seeds” 813; “731”; “fennel seeds” 814; “731”; “”; “spice mix“taco”” 815; “731”; “Soda” 816; “731”; “ginger juice squeezed” 817;“731”; “lemon juice” 818; “731”; “salt” 819; “731”; “citrate” 820;“731”; “grape sauce” 821; “731”; “sauce narsharab” 822; “731”; “ponzusauce” 823; “731”; “soy sauce” 824; “731”; “tomato sauce” 825; “731”;“chili sauce” 826; “731”; “Spices” 827; “731”; “sumac” 828; “731”;“thyme” 829; “731”; “cumin” 830; “731”; “Mediterranean herbs” 831;“731”; “French herbs” 832; “731”; “vinegar” 833; “731”; “balsamicvinegar” 834; “731”; “wine vinegar” 835; “731”; “white wine vinegar”836; “731”; “red wine vinegar” 837; “731”; “cherry vinegar” 838; “731”;“raspberry vinegar” 839; “731”; “rice vinegar” 840; “731”; “apple cidervinegar” 841; “731”; -”; “hops suneli” 842; “731”; “Savory” 843; “731”;“chutney” 844; “731”; “black pepper” 845; “731”; “black pepper peas”846; “731”; “dry garlic” 847; “731”; “sage” 848; “0”; “PREPAREDPRODUCTS” 849; “848”; “canned pineapple” 850; “848”; “canned artichokes”851; “848”; “Marinated artichokes” 852; “848”; “baguette” 853; “848”;“loaf” 854; “848”; “Bars of chocolate” 855; “848”; “meringue” 856;“848”; “biscuit” 857; “848”; “beans canned” 858; “848”; “bun” 859;“848”; “buns for hamburgers” 860; “848”; “broth” 861; “848”; “beefbroth” 862; “848”; “chicken broth” 863; “848”; “fish broth” 864; “848”;“Jam” 865; “848”; “Apricot jam” 866; “848”; “lingonberry jam” 867;“848”; “cherry jam” 868; “848”; “black currant jam” 869; “848”;“raspberry jam” 870; “848”; “blueberry jam” 871; “848”; “Wafer” 872;“848”; “canned cherry” 873; “848”; “Glaze” 874; “848”; “Dijon mustard”875; “848”; “croutons” 876; “848”; “marinated mushrooms” 877; “848”;“Demiglas apple” 878; “848”; “yeast” 879; “848”; “Jelly” 880; “848”;“leaven” 881; “848”; “marshmallows” 882; “848”; “crushed tomatoes injuice” 883; “848”; “pickled ginger” 884; “848”; “Cocoa” 885; “848”;“marinated cactus” 886; “848”; “Pickled capers” 887; “848”; “sourcabbage” 888; “848”; “sea ??kale” 889; “848”; “Kimchi” 890; “848”;“wafer cakes” 891; “848”; “gherkins” 892; “848”; “natural coffee” 893;“848”; “instant coffee” 894; “848”; “crackers” 895; “848”; “ChocolateCrumb” 896; “848”; “croissant” 897; “848”; “bouillon cubes” 898; “848”;“canned corn” 899; “848”; “marinated corn” 900; “848”; “pita” 901;“848”; “lanspik” 902; “848”; “ice” 903; “848”; “letcho” 904; “848”;“lasagna sheets” 905; “848”; “canned salmon” 906; “848”; “pickledonions” 907; “848”; “canned mandarins” 908; “848”; “marshmallow” 909;“848”; “hazelnut oil” 910; “848”; “sweet curd” 911; “848”; “yoghurt”912; “848”; “honey” 913; “848”; “honey in the comb” 914; “848”; “Mixginger” 915; “848”; “condensed milk” 916; “848”; “condensed milk boiled”917; “848”; “milk powder” 918; “848”; “pickled carrots” 919; “848”; “icecream” 920; “848”; “vanilla ice cream” 921; “848”; “chocolate ice cream”922; “848”; “salted cucumber” 923; “848”; “pickled cucumbers” 924;“848”; “pickled cucumbers” 925; “848”; “pecans” 926; “848”; “beet broth”927; “848”; “corn sticks” 928; “848”; “bread sticks” 929; “848”; “tomatopaste” 930; “848”; “Pasta Chocolate” 931; “848”; “pate” 932; “848”;“frozen dumplings” 933; “848”; “hot pepper pickled” 934; “848”; “cannedpeaches” 935; “848”; “Cookies” 936; “848”; “Biscuit” 937; “848”;“Cookies Savoiardi” 938; “848”; “chocolate cookies” 939; “848”; “pita”940; “848”; “supplements” 941; “848”; “tomatoes in juice” 942; “848”;“canned tomatoes” 943; “848”; “popcorn” 944; “848”; “prosciutto” 945;“848”; “gingerbread” 946; “848”; “mango puree” 947; “848”; “mashedpotatoes” 948; “848”; “tomato puree” 949; “848”; “apple puree” 950;“848”; “pickle cucumber” 951; “848”; “roll” 952; “848”; “Pickled beets”953; “848”; “pork jerky” 954; “848”; “sugar syrup” 955; “848”; “whippedcream” 956; “848”; “cream of coconut” 957; “848”; “malt” 958; “848”;“sorbet” 959; “848”; “barbecue sauce” 960; “848”; “sauce bearnez” 961;“848”; “bechamel” 962; “848”; “Worcestershire sauce” 963; “848”; “sauceDemiglas” 964; “848”; “” 965; “848”; “sweet and sour sauce” 966; “848”;“salsa” 967; “848”; “sweet sauce” 968; “848”; “chocolate sauce” 969;“848”; “berry sauce” 970; “848”; “asparagus soya” 971; “848”; “caramelchips” 972; “848”; “crushed crackers” 973; “848”; “tartlets” 974; “848”;“tahini” 975; “848”; “pasta for lasagna” 976; “848”; “dough for ravioli”977; “848”; “pizza dough” 978; “848”; “yeast dough” 979; “848”; “doughkataifi” 980; “848”; “shortbread dough” 981; “848”; “pastry dough” 982;“848”; “puff pastry” 983; “848”; “dough dry” 984; “848”; “filo pastry”985; “848”; “dried tomatoes” 986; “848”; “Tortilla” 987; “848”; “toast”988; “848”; “tofu” 989; “848”; “tuna fish oil” 990; “848”; “tuna cannedin its own juice” 991; “848”; “Tahini” 992; “848”; “ Rice Stuffing” 993;“848”; “Canned beans” 994; “848”; “white bread” 995; “848”; “toastbread” 996; “848”; “rye bread” 997; “848”; “sweet bread” 998; “848”;“black bread” 999; “848”; “rye bread” 1000; “848”; “corn flakes” 1001;“848”; “ciabatta” 1002; “848”; “tea Away” 1003; “848”; “potato chips”1004; “848”; “corn chips” 1005; “848”; “Marinated mushrooms” 1006;“848”; “chocolate corn balls” 1007; “848”; “Chocolate” 1008; “848”;“white chocolate” 1009; “848”; “bitter chocolate” 1010; “848”; “milkchocolate” 1011; “848”; “dark chocolate” 1012; “50”; “veal fillet” 1013;“57”; “beef fillet” 1014; “848”; “sauce for soups “Bright udon”” 1015;“296”; “Lemon” 1016; “217”; “Carrots” 1017; “217”; “Tomato”

TABLE D Types of Cuisine and Dishes Types of Cuisine 1 Abkhaz cuisine 2Australian cuisine 3 Austrian cuisine 4 Azerbaijan cuisine 5 Albaniancuisine 6 Algerian cuisine 7 American cuisine 8 English cuisine 9 Arabiccuisine 10 Argentine cuisine 11 Armenian cuisine 12 Bashkir cuisine 13Belarusian cuisine 14 Belgian cuisine 15 Bulgarian cuisine 16 Bosniancuisine 17 Brazilian cuisine 18 Hungarian cuisine 19 Venezuelan cuisine20 Vietnamese cuisine 21 Greek cuisine 22 Georgian cuisine 23 Danishcuisine 24 Jewish cuisine 25 Israeli cuisine 26 Indian cuisine 27Indonesian cuisine 28 Jordanian cuisine 29 Iraqi cuisine 30 Iraniancuisine 31 Irish cuisine 32 Icelandic cuisine 33 Spanish cuisine 34Italian cuisine 35 Cambodian cuisine 36 Canadian cuisine 37 Cypriotcuisine 38 Chinese cuisine 39 Colombian cuisine 40 Korean cuisine 41Creole cuisine 42 Costa Rica cuisine 43 Latvian cuisine 44 Lebanesecuisine 45 Libyan cuisine 46 Lithuanian cuisine 47 Macedonian cuisine 48Malaysian cuisine 49 Moroccan cuisine 50 Mexican cuisine 51 Moldaviancuisine 52 Mongolian cuisine 53 German cuisine 54 Dutch cuisine 55Zealand cuisine 56 Norwegian cuisine 57 Ossetian cuisine 58 Pakistanicuisine 59 Palestinian cuisine 60 Panamanian cuisine 61 Peruvian cuisine62 Polish cuisine 63 Portuguese cuisine 64 Romanian cuisine 65 Russiancuisine 66 Serbian cuisine 67 Singaporean cuisine 68 Syrian cuisine 69Slovak cuisine 70 Slovenian cuisine 71 Thai cuisine 72 Tatar cuisine 73Tibetan cuisine 74 Tunisian cuisine 75 Turkish cuisine 76 Turkmencuisine 77 Ukrainian cuisine 78 Philippine cuisine 79 Finnish cuisine 80French cuisine 81 Croatian cuisine 82 Montenegrin cuisine 83 Czechcuisine 84 Chilean cuisine 85 Chuvash cuisine 86 Chukotka cuisine 87Swedish cuisine 88 Swiss cuisine 89 Scottish cuisine 90 Ecuadoriancuisine 91 Estonian cuisine 92 Japanese cuisine 93 Raw food diet 94Estonian cuisine 95 Japanese cuisine 96 Raw food diet 1 Types of Dishes1 Snacks 2 Salads 3 Entrees 4 Main Dishes 5 Desserts 6 Drinks 7 Saucesand marinades 8 Baking 9 Semimanufactures and preservatives

TABLE E List of Robotic Food Preparation System (One Embodiment) LevelSys Responsible Major of No Category Category System Party(s) ChallengesCompletion Notes 01 Hardware Robot Hands Productionization, Robustness,Cost, Weight 02 Hardware Robot Arms 03 Hardware Robot Armature Rails 04Hardware Capture/ Dynamic 3D Vision Training System 05 Hardware Capture/Data Input Training 06 Hardware Capture/ Editing System Training 07Hardware Kitchen Cabinets Module 08 Hardware Kitchen Fixtures Module 09Hardware Kitchen Lighting with ability to Module computer-operating mode10 Hardware Kitchen Protection/Safety Module Screen with ability tocomputer-operating mode 11 Hardware Kitchen Appliances Module 12Hardware Kitchen Automatic Storage Module device with ability tocomputer-operating mode 13 Hardware Kitchen Automatic modular Moduledispenser for flowing, liquid ingredients and water with ability tocomputer-operating mode Hardware Kitchen Freshness ingredients Moduleanalytical device Hardware Kitchen Built-in electronic scales Module (inthe tabletop) with ability to computer- operating mode 14 HardwareKitchen Cleaning Module 15 Hardware Kitchen Waste Disposal ModuleHardware Kitchen Multi-functional Module professional steam- oven withability to computer-operating mode Hardware Kitchen Multi-functionalModule professional kitchen processor with ability to computer-operatingmode Hardware Kitchen Top-loaded dishwasher Module with ability tocomputer-operating mode Hardware Kitchen Professional Stove with Moduleturning control regulators/buttons operated with ability tocomputer-operating mode Hardware Kitchen Standard dimension Modulelayout Hardware Kitchen Anti-wieting, smoke, Module steam ventilationsystem autonomous or be connected to the duct with ability tocomputer-operating mode Hardware Kitchen Kitchen sink with tap Modulewith ability to computer-operating mode 16 Hardware Control/ CPU PowerModules 17 Hardware Control/ I/O Touchscreen Power Modules 18 HardwareControl/ Power Supply Power Modules 19 Hardware Accessories Utensils 20Hardware Accessories Food Containers/Cartridges 21 Software Robot OSModule Hardware Kitchen Professional Stove with Module turning controlregulators/buttons operated with ability to computer-operating mode 21Software Robot OS Module 22 Software Robot Apps Module 23 Software Robothand firmware Module 24 Software Robot Arm firmware Module 25 SoftwareRobot Rail Control Module 26 Software Capture/ OS Training 27 SoftwareCapture/ apps Training 28 Software Capture/ Vision Training 29 SoftwareCapture/ Data Input Training 30 Software Capture/ Editing SystemTraining 31 Software Kitchen OS Module 32 Software Kitchen App Module 33Software Kitchen Controller Module Protection/Safety 34 Software KitchenController, Appliances Module 35 Software Kitchen Controller, StorageModule 36 Software Kitchen Controller, Cleaning Module 36 SoftwareKitchen Controller, Steam-oven Module 36 Software Kitchen Controller,Kitchen Module Processor 36 Software Kitchen Controller, DishwasherModule 36 Software Kitchen Controller, Stove Module 36 Software KitchenController, Ventilation Module system 36 Software Kitchen Controller,Lighting Module 37 Software Kitchen Controller, Waste Module 37 SoftwareKitchen Controller, Tap Module 37 Software Kitchen Controller,Dispensing Module device 37 Software Kitchen Controller, Scales Module37 Software Kitchen Controller, Freshness Module Indicator 38 SoftwareControl/ OS Power Modules 39 Software Control/ I/O Touchscreen PowerModules 40 Software Control/ Control Apps Power Modules 41 Other FoodFood Recipe Development 42 Other Food Food Container Prep 43 Other FoodFood Order/Delivery 44 Other Logistics Safety/Regulatory 45 OtherLogistics Sales/Distribution 46 Other Logistics Installation/Maintenance47 Other Logistics Packaging/Shipping Container 48 Other LogisticsReturn Management 49 Other Logistics Technical Training 50 OtherLogistics Manuals 51 Other Logistics Warranty 52 Production Robot 53Production Kitchen Module 54 Production Integration/ Shipping 55Production 56 Production 57 Production 58 Production 59 Production 60Production

The present invention has been described in particular detail withrespect to possible embodiments. Those skilled in the art willappreciate that the invention may be practiced in other embodiments. Theparticular naming of the components, capitalization of terms, theattributes, data structures, or any other programming or structuralaspect is not mandatory or significant, and the mechanisms thatimplement the invention or its features may have different names,formats, or protocols. The system may be implemented via a combinationof hardware and software, as described, or entirely in hardwareelements, or entirely in software elements. The particular division offunctionality between the various systems components described herein ismerely example and not mandatory; functions performed by a single systemcomponent may instead be performed by multiple components, and functionsperformed by multiple components may instead be performed by a singlecomponent.

In various embodiments, the present invention can be implemented as asystem or a method for performing the above-described techniques, eithersingly or in any combination. The combination of any specific featuresdescribed herein is also provided, even if that combination is notexplicitly described. In another embodiment, the present invention canbe implemented as a computer program product comprising acomputer-readable storage medium and computer program code, encoded onthe medium, for causing a processor in a computing device or otherelectronic device to perform the above-described techniques.

As used herein, any reference to “one embodiment” or to “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiments is included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification are not necessarilyall referring to the same embodiment.

Some portions of the above are presented in terms of algorithms andsymbolic representations of operations on data bits within a computermemory. These algorithmic descriptions and representations are the meansused by those skilled in the data processing arts to most effectivelyconvey the substance of their work to others skilled in the art. Analgorithm is generally perceived to be a self-consistent sequence ofsteps (instructions) leading to a desired result. The steps are thoserequiring physical manipulations of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical, magneticor optical signals capable of being stored, transferred, combined,compared, transformed, and otherwise manipulated. It is convenient attimes, principally for reasons of common usage, to refer to thesesignals as bits, values, elements, symbols, characters, terms, numbers,or the like. Furthermore, it is also convenient at times to refer tocertain arrangements of steps requiring physical manipulations ofphysical quantities as modules or code devices, without loss ofgenerality.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that, throughout the description, discussionsutilizing terms such as “processing” or “computing” or “calculating” or“displaying” or “determining” or the like refer to the action andprocesses of a computer system, or similar electronic computing moduleand/or device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission, or displaydevices.

Certain aspects of the present invention include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present inventioncould be embodied in software, firmware, and/or hardware, and, whenembodied in software, can be downloaded to reside on and be operatedfrom different platforms used by a variety of operating systems.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, application specific integratedcircuits (ASICs), or any type of media suitable for storing electronicinstructions, and each coupled to a computer system bus. Furthermore,the computers and/or other electronic devices referred to in thespecification may include a single processor or may be architecturesemploying multiple processor designs for increased computing capability.

The algorithms and displays presented herein are not inherently relatedto any particular computer, virtualized system, or other apparatus.Various general-purpose systems may also be used with programs inaccordance with the teachings herein, or it may prove convenient toconstruct more specialized apparatus to perform the required methodsteps. The required structure for a variety of these systems will beapparent from the description provided herein. In addition, the presentinvention is not described with reference to any particular programminglanguage. It will be appreciated that a variety of programming languagesmay be used to implement the teachings of the present invention asdescribed herein, and any references above to specific languages areprovided for disclosure of enablement and best mode of the presentinvention.

In various embodiments, the present invention can be implemented assoftware, hardware, and/or other elements for controlling a computersystem, computing device, or other electronic device, or any combinationor plurality thereof. Such an electronic device can include, forexample, a processor, an input device (such as a keyboard, mouse,touchpad, trackpad, joystick, trackball, microphone, and/or anycombination thereof), an output device (such as a screen, speaker,and/or the like), memory, long-term storage (such as magnetic storage,optical storage, and/or the like), and/or network connectivity,according to techniques that are well known in the art. Such anelectronic device may be portable or non-portable. Examples ofelectronic devices that may be used for implementing the inventioninclude a mobile phone, personal digital assistant, smartphone, kiosk,desktop computer, laptop computer, consumer electronic device,television, set-top box, or the like. An electronic device forimplementing the present invention may use an operating system such as,for example, iOS available from Apple Inc. of Cupertino, Calif., Androidavailable from Google Inc. of Mountain View, Calif., Microsoft Windows 7available from Microsoft Corporation of Redmond, Wash., webOS availablefrom Palm, Inc. of Sunnyvale, Calif., or any other operating system thatis adapted for use on the device. In some embodiments, the electronicdevice for implementing the present invention includes functionality forcommunication over one or more networks, including for example acellular telephone network, wireless network, and/or computer networksuch as the Internet.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. It should be understood thatthese terms are not intended as synonyms for each other. For example,some embodiments may be described using the term “connected” to indicatethat two or more elements are in direct physical or electrical contactwith each other. In another example, some embodiments may be describedusing the term “coupled” to indicate that two or more elements are indirect physical or electrical contact. The term “coupled,” however, mayalso mean that two or more elements are not in direct contact with eachother, but yet still co-operate or interact with each other. Theembodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof are intendedto cover a non-exclusive inclusion. For example, a process, method,article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

The terms “a” or “an,” as used herein, are defined as one or more thanone. The term “plurality,” as used herein, is defined as two or morethan two. The term “another,” as used herein, is defined as at least asecond or more.

An ordinary artisan should require no additional explanation indeveloping the methods and systems described herein but may find somepossibly helpful guidance in the preparation of these methods andsystems by examining standardized reference works in the relevant art.

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of the abovedescription, will appreciate that other embodiments may be devised whichdo not depart from the scope of the present invention as describedherein. It should be noted that the language used in the specificationhas been principally selected for readability and instructionalpurposes, and may not have been selected to delineate or circumscribethe inventive subject matter. The terms used should not be construed tolimit the invention to the specific embodiments disclosed in thespecification and the claims but should be construed to include allmethods and systems that operate under the claims set forth hereinbelow. Accordingly, the invention is not limited by the disclosure, butinstead its scope is to be determined entirely by the following claims.

What is claimed and desired to be secured by Letters Patent of theUnited States is:
 1. A smart kitchen, comprising: a kitchen module,comprising: a kitchen furniture, and one or more smart appliances; oneor more sensors; a display for displaying one or more controllingfunctions in the kitchen module; and one or more processors forcontrolling the one or more smart appliances; wherein during a firstmode, the one or more sensors for recording a first set of data in arecipe file for controlling one or more smart appliances, and during asecond mode, the one or more sensors for receiving a second set of datafor subsequent corrective or feedback during execution of the recipefile by the one or more smart appliances; and wherein the one or moreprocessors executes the recipe file for controlling the one or moresmart appliances, each smart appliance executing one or more foodpreparation operations based on one or more graphical curves from therecipe file, the one or more processors generating a command forcontrolling the one or more smart appliances from the recipe file orfrom the user, the one or more graphical curves comprising one or moredata over a time period.
 2. The smart kitchen of claim 1, wherein theone or more sensors is integrated in the kitchen module.
 3. The smartkitchen of claim 1, wherein the one or more sensors is located outsideof the kitchen module.
 4. The smart kitchen of claim 1, wherein the oneor more sensors is integrated in the one or more smart appliances. 5.The smart kitchen of claim 1, wherein the information comprises data andinstructions.
 6. The smart kitchen of claim 1, further comprising one ormore smart containers for storing ingredients.
 7. The smart kitchen ofclaim 1, further comprising one or more dispensing devices for dosingingredients.
 8. The smart kitchen of claim 1, further comprising arecipe Library database for storing a plurality of recipe files forexecution by the one or more smart appliances within the kitchen module,each of the recipe files includes one or more cooking operations with astart time and an end time.
 9. The smart kitchen of claim 1, wherein theone or more processors receives data from a chef and record the data ina recipe file.
 10. The smart kitchen of claim 1, where the one or moreprocessors in the kitchen module coupled to a network for controllingthe one or more smart appliances and for communicating informationbetween the kitchen module and the graphical or voice user interface,the interface receiving a voice activation command from a user, theinterface generates generating a voice-based information to the user.11. The smart kitchen of claim 1, where the kitchen module comprises oneor more robotic arms and one or more robotic end effectors, theprocessor executing the recipe file in which the one or more roboticarms and the one or more robotic end effectors operating the one or moresmart appliances.
 12. The smart kitchen of claim 1, wherein theprocessor generates a visual graphical representation of the smartkitchen.
 13. A method for operating a kitchen module in a smart kitchen,comprising: providing a kitchen module, the kitchen module including akitchen furniture, one or more smart appliances, and one or moresensors; during a first mode, recording by one or more sensors a firstset of data in a recipe file for controlling the one or more smartappliances, and during a second mode, the one or more sensors receivinga second set of data for subsequent corrective or supportive feedbackduring the recipe execution, or to define position and/or orientation ofa standard object in the kitchen module, or to define a shape, a size, atype, a position, or an orientation of a non-standard object in thekitchen module, the one or more sensors sensing the non-standard objectwith respect to a shape, a dimension, an orientation or a positioninformation for making a real-time adjustment; displaying one or morereal-time smart appliances statuses upon receiving a chef or usercommand via a graphical or voice user interface for selecting the recipefile for recording or execution, displaying recipe execution feedback,visualizing object position and orientation in the kitchen, or visualizecontrolling functions of the one or more smart appliances, or anycombinations thereof, in the kitchen module during recipe recording andexecution; detecting by one or more processors operational states of theone or more smart appliances and operating the one or smart appliances;and controlling the one or more smart appliances by e one or moreprocessors in the kitchen module coupled to a network, the one or moreprocessors communicating information between the kitchen module and thegraphical or voice user interface; wherein the one or more processorsexecutes the recipe file for controlling the one or more smartappliances, each smart appliance executing one or more food preparationoperations based on one or more graphical curves from the recipe file,the one or more processors generating a command for controlling the oneor more smart appliances from the recipe file or from the user, the oneor more graphical curves comprising one or more data over a time period.14. The method of claim 13, during an execution mode, further comprisingexecuting the recipe file for controlling the one or more smartappliances and checking on the real-time status of the one or more smartappliances.
 15. The method of claim 14, wherein the one or moreprocessors informs a user via a visual interface or a voice interface onthe real-time status of the one or more smart appliances.
 16. The methodof claim 14, wherein the one more processors assists a user with foodpreparation via a visual interface or a voice interface based on thereal-time status data collected from the one or more smart appliances.17. The method of claim 14, during the recording mode, furthercomprising recording the recipe file from real-time sensor data, thereal-time sensor data including sensor data and timing data.
 18. Themethod of claim 17, wherein the real-time sensor data comprises one ormore video sensors, one or more microphone sensors, one or moretemperature sensors, and/or sensor data from the one or more smartappliances.
 19. The method of claim 13, further comprising retrievingthe file from a cloud database.
 20. The method of claim 13, wherein thedisplaying step comprises displaying a real-time virtual kitchen modelvia a visual interface.
 21. The smart kitchen of claim 1, wherein duringthe second mode, the one or more sensors receiving the second set ofdata to define a position and an orientation of a standard object in thekitchen module, or the one or more sensors receiving the second set ofdata to define a shape, a size, a type, a position, or an orientation ofa non-standard object in the kitchen module, the one or more sensorssensing the non-standard object with respect to its shape, dimension,orientation or position information for making a real-time adjustment.22. The smart kitchen of claim 1, wherein the step of the graphical orvoice user interface for operating the kitchen module or obtainingreal-time information from the one or more objects in the kitchen modulecomprises (1) receiving a chef or user command to select the recipe forrecording or execution (2) displaying one or more real-time smartappliances statuses, (3) displaying recipe execution feedback, (4)visualizing one or more objects, a respective objection position and/ora respective object orientation in the kitchen, or (5) controlling theone or more smart appliances, or any combinations thereof.
 23. The smartkitchen of claim 1, wherein the kitchen module comprises one or moreactuators to reallocate one or more objects in the kitchen module,including one or more shelfs, one or more cookwares, or one or moresmall appliances.
 24. The smart kitchen of claim 1, wherein the one ormore smart appliances comprises a pan having a plurality of cookingzones, each cooking zone associated with a respective control and arespective sensor, the respective control measuring an adjusting thetemperature of a respective cooking with the respective sensor, the panreplicating the chef creating recipe process by controlling andadjusting one or more desired temperature curves over time in theplurality of cooking zones.
 25. The smart kitchen of claim 1, furthercomprising one or more ingredient storages having one or more withsensors for receiving real-time sensor data to determine ingredientproperties in the one or more ingredient storages.
 26. A smart kitchen,comprising: a kitchen module, comprising: a kitchen furniture, a worktoparea, one or more smart appliances, and one or more sensors; one or moresensors; a display for displaying one or more controlling functions inthe kitchen module; and one or more processors in the kitchen modulecoupled to a network for controlling the one or more smart appliancesand for communicating information between the kitchen module and thegraphical or voice user interface; wherein during a first mode, the oneor more sensors for recording a first set of data in a recipe file forcontrolling one or more smart appliances, and during a second mode, theone or more sensors for receiving a second set of data for subsequentcorrective or supportive feedback during execution of the recipe file bythe one or more smart appliances; and wherein the one or more smartappliances comprises a smart oven communicatively coupled to anoperating control unit, the smart having one or more temperature sensorsand/or one or more humidity sensors, the smart oven replicating the chefcreating recipe process with the sensor curves of a particular foodrecipe in which the operating control unit adjusts the temperature andhumidity of the smart oven by real-time data received from the one moretemperature sensors and/or one or more humidity sensors.
 27. A smartkitchen, comprising: a kitchen module, comprising: a kitchen furniture,a worktop area, one or more smart appliances, and one or more sensors;one or more sensors; a display for displaying one or more controllingfunctions in the kitchen module; and one or more processors in thekitchen module coupled to a network for controlling the one or moresmart appliances and for communicating information between the kitchenmodule and the graphical or voice user interface; wherein during a firstmode, the one or more sensors for recording a first set of data in arecipe file for controlling one or more smart appliances, and during asecond mode, the one or more sensors for receiving a second set of datafor subsequent corrective or supportive feedback during execution of therecipe file by the one or more smart appliances; and wherein the one ormore smart appliances comprises a charcoal grill communicatively coupledto a control unit and an electric ignition system, the charcoal grillincluding one or more temperature sensors and one or more humiditysensors, charcoal grill replicating the chef creating recipe processwith the sensor curves of a particular food recipe in which the electricignition system modulates electric power to the charcoal grill withreal-time data received from the one more temperature sensors and/or oneor more humidity sensors.
 28. The method of claim 13, wherein thenon-standard object comprises an ingredient that may have a differentsize, a different dimension, a different shape, a different color, adifferent texture, a different weight, and/or any combination thereof.29. The method of claim 13, wherein the standard object including thename of the standard object, a predetermined size, a predetermineddimension, a predetermined color, a predetermined shape, a predeterminedtexture, a predetermined weight, a standard position, a standardorientation, a virtual three-dimensional model of the standard object,and/or any combination.
 30. The method of claim 13, wherein the kitchenmodules comprises a non-standard kitchen module, the standard objectplaced in a different position and/or in a different orientation. 31.The smart kitchen of claim 1, wherein the one or more processors arelocated in the one or more smart appliances.
 32. The smart kitchen ofclaim 1, wherein the one or more processors are located in the kitchenmodule.